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. - . Chapter 1 

I V 

BACKGIDUND AND PUWOSES OF THE STUDY ' ■ 

* • ■ * 

^Mriaaa isnlveTdlties are under growing pressure to evuluate 
ud ev^ ta Justify the cent inuW existence of their doctoral 
pTOgrama* These pressures are generated, principally , by concerns 
abouc persistent financial shortagea and apparent degyee over- 
production as well as by t^e increasing authority of state 
educational coordinating bcaras and other eKternal agencies. 
Itistitations and states are looking for fair and depen'd^le ways 
to evaluate graduate progr^s in order to use their resources 
most productively I departments want Infffimation that will help 
them adapt pi^ograins to changing needs | prospective students need 
more Information about the characteristics ^nd emphasias of specific 
programs for Imprpved attendance decisions* All^of these groups 
are concerned about whether or not the quality M doctoral programs 
will be maintained in the face of these pressures* 

The project on the Dlmensdons of Quality In Doctoral Education 
Is a 'response to ^hese needs and concerns. It developed fyom the \ 
prMisa that the many aspects bf doctoral progrOTs thafr contribute 
to educational excellence a»e represented only partially and 
laperf^rtly by any single rating or index of progrm quality* Thp 
prtaary goal of the study was to describe programs in ways that 
would Mcdurage and support their itftlvidual excelle^e «id at the 
same tine l^rove the information available for decisions by policy 
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miners and prospective students. In order to accomplish this goal 
It was nmcmm^mty to Identify the program .characteristics moat 
closely assQclated with 'Judgments fbout educational quality ^ 
. davelap sellable m^asi^e^ ot Lli^^m chaLacteri^tlc^ » #nd to select 
a inanageable nwber ©f them for prof llf s of .the wa^s in which one 
pragram 'compares wl tu others in the samt. discipline* This, report 
describes such activities. ' ^ ^ , 

Background , ■ * 

This study of multiple criteria of progr§im. eKcellen£^ grew 

^ out of concern mong a number of graduate school deans about ,the 

. limitations of reputational ratings as the only .widely ackn^ledged 

Indicator of doctoral program quality. Ratings of the reputation 

of a program among faculty embers or other professl^als In the 

same field have a place in program evaluation; but they are not 

very helpfi^ to those who may be seeking to Improve their programs » 

,are highly related to progranj slse and visibility^ an tf only 

occasionally reflect recent changes (elti.er good or bad) in a 
* r - 

program. Many activities cont^bute to program excellence'; 

therefore I multiple criteria are needed to Judge the leval of a 

program* achievement * , / 

Interest In ways to measure and to evaluate many of the 

r different p^^rogram characteristics that are associated with 

educational excellence also rn fleets an increasing demand for 

program aceount^llity. Particularly at the graduate level, 

state agencies and national councils, as weij^as individual 
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Inscltutlans, ara calling for progrm reviews that .consider mission, 
naad, aos£, and productivity as well as quality (e.g. , Fleming, 1973| 
Education CoBo^ssion ©f the States, igTsTN^tional Board on Graduate 
fiducation, 1975)^. Tlim Importance of quality is epiphaslstd In the 
rhetbric; but the review procedurep that lire described or suggested 
almost always concentrate on easily available quantitative measurea 
•such as ni^er of students * number of degrees granted, or national 
raputational rating in the field, rather than attemp^ to come to 
grips with what "quality" really means in graduate eduction, or 
with some of the ways in which it might be evaluated. 

The major effort to evaluate the quality of doctoral programs 
in recent years has been the collection of prestige or reputational 
ratings by the . arican .Council uii Education in 1964 and 1969 
(Cartter, 1966; Roose a Andersen, 1970). In addition to direct 

use by the higher education comiunityj these rat iigs have been ufesd 

* /' 

as criterion measutes in several efforts to evaluate sAlected i 
characteristics, of Ph.D. programs (e.g., Drew, 1975; Elton a Rogei^, 
1971; Elton. & Host, 1972; Hagstrom, 1971; National Science Board, 
1969; ^el and Lamson, 1972) and to select graduate progrw^ of 
varying quality for more intensive itudy (e.g., Dresseli Johnson, 
. h Marcup, 1970; Heiss, 1970). These studies , and others, identified 
a ntMber of structural program ^pr^acteristics associated wAth the 
reputational ratings, faliing generally into five categories i 
<1) salsctiyity (e*g. , nuii±>er of sturferit natiofial fellowship \ 
holders, a^er of students from highly selective undergraduate , 



eciillegaii, i^tmbsr of faculty with PhiD.'j from hlgh-prestlga progr^^^ 
(2) siza Mid rasoui eaR (a.g. p nimb^.r of studanti, nitfabsr at faculty 
mMbersp average faculty compensation , wotmt of financial support ' 
to students) • progTOT emphaals (e*g*, number of doctoral degress 
airarded In delation to number of baccalaureate degrees awarded , 
part^tlM vs. fUll'-tlme studpQti» priorities among nea^lng/researdi/ 
service); (4) produ^lvlty '(e.g« , n^^er of degrees grmted, faculty 
publications s elapsed time to the degree) and (5) self-evaluations » 
(e.g.y inerale^ satlsf act^onp peer appraisal). Only the larger 
studies by Dressel, et a.L. » and HelsSi in which selected- prograias 
wete visited and studied in some depth, made any attempt to relate 
program procaases to quality by including pbservational and survey 
information. ^ 

More recently, a number of chackliats for the assessment €^ 
academic quality or, the internal review of graduate progrois have 

been drawn up (e.g., Balderston-, 1974| Council of Graduate Schools 

_ _ _ ' - 

Task Force on Internal Review of Graduate Programs , 1974; McHlchaelp 

1973) « These also tend to emphasize the input quality of students 

\ ' ■ ■ 

find faculty ^.embers, .the resear^ productivity of the faculty, the 

degree productivity of studenti, and physical facilities and other 

reapurces^ though there la somewha^more attention to the 

educational process than In the research studies « Taken together » 

however s the literature suggests a bewildering array of graduate 

vprogm characteristics that one might investigate In relation to 

educational quality , but provides very little guidance about the 



relative Imporcwai of the various characterlatlca . The literature 
ajL#Q tmds to suggest, ways measur^g only the most obvloas 
straatural elments . such as size and^degree productivity. 

^.Ihis was the iltuatlon ^.n the spring of 1973 when the Gradate 
tee^rd EscMdnatlotia Board conmlsslbned a smalJ-fcale survey of 
gridumte dean^ In order to Identify a manageable llgt of program 
chr.racterlstles that were most Important to educational quality 
4nd some acceptable ways of ffieasurlng them (Clark, 1973) f &li:5y 
panel members were selected to represent* a cross section of Ph.D,- 
grmtlng universities according to geographic location, prestige* 
slEe, and coatrol. The study wa€ designed to provide four kinds 

of Informations . 

1, '^he extent of Agreement among graduate deans about Fh.D, 
program characteristics mofe't Important ^© Judgments about 
quality; 

2, variations in the Importance of characteristics depending 
upon the major purpose of a Ph*D, pro^am| 

3, the judged adequacy or acceptability of poaslble measures 
for eaeh characteristic; and ^' 

4* Jthe current avallabUlty of various kinds of Information. 
It was thought that the deans* responses to these questions would 
be of imadlate Interrtt to membsfrs of the graduate school ^ 
eoommltyp and '^hat they also would help to specify areas for 
primary attention in later research* 



The lljts 6^f program characterlstlrd and pots ' le measures that 
vsrs subaltttd to the dams were complied from all p^^^ible sources, 
Inaluding the research literature, task force reports, Individual 
unl^rslty naterlals and reports* and consultation with experts In 
the field of graduate education. A very high response jate on 
fvo questionnaires s,80% to 90%) In a brief time attested to the 
interest of these graduate deans In the torlc of quall^« 

On the basl^ 'of the deantf' ratings , the nu^dser of possible 
« program characteristic^ Import^fi to Judgments about educfMonal 
quality was reduced by a third, from 63 to 42 Items, arranged 
under the general headings of faculty menA^ers, studencs, program 
resources and program opc ration « A total ot 290 possible measurea - 
were then proposed for the 42 "important" or "very Important" 
char aeterls ties* 'Ratings by the deans^cUueed this nUmber by more 
than half, leaving 133 possible meaauros rated "gcod** or J'very good" 
aa Indicators of 'the quality of the most ^^^drtant program 
characteristics. 

A review of the acceptable indicators s\»ggested it leas' t 
seven potantial sQurces of information important to Judgments about 
educational quality, ^out a third of the endorsed measures were 
in the tuLm of statistical and financial Information from program 
and instltuticnal records. Other desirable sources of information 
Included faculty members, students, recent ^lusmi, progrMn chalnnen 
and deans, employers of graduates, and visiting tems of expertsi 
Panel meters often endors^ed more than one source fop a glvfen item 
of Information. 



This *simary list of important program characteristics, and 
their acceptable Indicators ^ao then reviewed by the steering 
cMttifctes that supervised the panel survey^ and otners to eliminate 
characteristics or meestir^a which, rithuugh desirable, ©ight be 
too expensive, too unreliable, 'or In other ways particularly 
f^*iff4cult to obtrln. For instance, though ratings by visiting 
paiels of experts were highly endorsed by the deans for many 
characteristics, the reviewers elect jd to eliminate this source 
of infbrmatton from the next effort (the research sunmarlged in 
this report) becaus^^ of the high cost of obtaining reliable 
information from visiting, teams f or ,a number of programs in a^ 
sshort time. Similarly, Information from the Bmplgyers of alumni 
was recognised as dpslrable but Impractical for any large-scale 
study This review procedure resulted In a reflnRd list, cf 
30 program characteristics that graduate deans chougtit were closely 
related to educational quality and that appeared to be Mienable to, 
measurement. These c! aracterlstics are* listed In Table 1.1, 
toother with one or mor endorsed sources of Information for eicb 



-Elg^t graduate dedns appointed by the Graduate Record 
Exmdnatl'ons Board and by the Council of Graduate Schools. 
Appointed by GBEB wpre Wllllara Burke, Arizona State Universleyi 
Betnatd Harleeton, Tufta Unlveraltyi Robert MacFarland, Vnlvereity . 
of Missouri at Rollai and Donald Taylor, Yale University. Mfaibera , 
•appointed by COS included Mary Evelyn Huey, Texas Woman's Unlyersltys 
Philip Kubsimsky, Boston University; Charles Lester, Biiory UnlverBlty 
md Joaeph McCarthy, UnlveralCy of Washington. Michael Pelciar, 
OnlverBity of Maryland and chairman of the GRETJ, and j.-Boyd Page, 
PreBident of CGS, were ex of f leig meters of the co™ittee. 

' s 

-Botli of these sources of Information may, of course, be 
'^valTAle and valuable as parts of Individual puogrmi evaluations^^ 
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Table A.l 



tVQgtm ChrnvmitrnttBttcm Judged import&nC to (^lallty ana 
Sobs Aeoipcabla Sources of Information About Tnam 



Sources of Information 



Chfract^riJtlcs 



Records 



Faculty Btudent Alimnl 
Que8» ^ Ques. Quas. 



5, 
7. 



10. 



11. 



^ Xr- Acadmle trid-slng % 
lnvMryh accivlty ' ' , X 

Rsiafit^ produotlvlty 
Tatihlag effect ivenes& ^ 
Co&ee^ fdr student development 

and welfare 
laiiFplvraftat ptogrm affffl^^s 
Group norale or esprit 

STUDDITS 

8^ AeadeMc ability at entrance X 
9m Aohlevaients/krowledge/skills at 

tisa of degree eo^letloni X 
Profasalonal acooarpliahrents ^ 

of graduates ' 
Judgp^ts about program 'lallt/ 
12 # Satisfactions with v&rloua aspects 
of progifm 

13. Group laorale or esprit 

RESGtonaS 

14. Flnraclal support" Internal dmA external 

(Inaludlng edueatlon and general t 
flnai^clal aid for students , research) X 
Library , ^ 

Laboratory equlpraent and facilities 
Computer facilities I 

OPlRAtlOlE 

18 Furposes of the program X 
Coursa and pregrra offarlngs 
Adalsslons policies . X 

Kaculty welfare X 
Evaluation "^f student progress 
Progrm leadership Bnd decision-making 
^24. Job placement of graduates ' K 

25. Advisment of studanti' 

Student-f acult^: Interaction 
In^tarnshlps I a§3l|&antihlps and other 
dpportu&ltlas for relevant student 
escperlances . _ X 

28. >iagraa requirements 'X 

29. Relatlpfishlps with co^ate program.'i 

30. Efficiency of degree production X 



15* 
16. 
17/ 



19. 

20. 
21. 
22; 
23. 



26. 
27. 



X 

X 



X 

X 



X 
X 

X 

X 
X 

X 

X 



x/^ 



X 

X 



X 
X 



X 
X 



X 
X 



X 
X 



X 
X 
X 
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eharaQterlstlc, More detslied Infotmatlon about the deans' ratlixgs 
of t'^n diaraaterlsfles and measures can be found In Appendix C. 

Tha major purposa.of the research project that is describeo tu 

J 

this repokt was to gain's better understanding of dostoral program 
educational quality y Jeveloping and field testing measures for 
importTOt quality-related program characteristles. I^e study 
concentrated on questionnaire data from progrm participants-- 
students, faculty, and recent alumi—and Infomation froq depart- 
mental records » since these sources ware endorsed by the expert- 
panel and were also readily accessible. The study was exploratory 
in nature* It was limited to doctoral^ progrtfns in one discipline 
^ frOT each of the areas of natural science (chOTistry) , social 

sciencl (psychology) , and himanities (history) . It also made an 
' effort: to consider program evaluation in the context of such dltfar- 
Ing program- purposes as the preparation of scholarly researchers, 
cQil^g^ teachers, and other professional practitioners, ^ 

Three more specific sets of research questions guided the . 
design tod conduct of the' study i 

1. Cm reliable data concerning important program characteristics 
y be obtained in forts that are consistent from progrm to 

progrTO? What kinds of infomation are most accessibly, an4 
wh»C kinds are hard to obtain? The basic issue here Is on^ of 

v' \ 

:■ • neasurfflwat feasibility and reliability, with «nphaals on 

pro-am indicators that are already available obtainable , 
without the use of site visxtss visiting teams, or intervl^s. 
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An evaluation of consistency is important because It is 
essential to the comparison of. th4 characteristics of one 
progr^ with the characteristics of others , either within 
the same dlfictpllne ^rrosR Mnlv^rRi r f i^r <^ ArrnR?^ d1srlpTfn#« 
within tha. ssmm university* ' 

Are some program charar terlft Ics Important In judging quality 

only in certain disciplines? Even though the same measures ^ 

can be used in different disciplines ^ do patterns of reiponse 

differ by discipline? Do the interrelationships among 

* 

Indicators vary by discipline? The primary research interest 

here is one of generallzabillty . If measures appear to 

operate similarly in fields as diverse as chemistry , history^ 

and paychologyj then theme is reason to believe that they 

wouldL operate similarly In doctoral programs In a ^^rlety of 

ot^er fields as well. On the other hand, measures that appea 

to work quite differently in these three dls^pllnes may need 

to be tried out in other fields before the generallzabillty 
of their use can be estimated* 

Can '^f^>qsureF of qn^l i ^y-re*i ntPd p^ngram characf:eTlF tlc^ 
eontribute to the improvement of individual progrtoa? Can 
they contribute to improvements in university- aad state- 
level systems of progr^ review? Can they improve the " 

information available to prospective graduate students? 

* 

Primary concerns here are data validity^ and the uses that 
are, made of program descriptors when they are availably to 



< 
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pTQgrra managers and higher-education decision makers. Some 
^of the research efforts, such as analyses to evaluate the 
interrftlatlonships ampng variables and the extent of agreement 
when different sources report on the same characteristic of a 
pragrim, assess the accuracy of the measures. Other aspects 
of the raaearch, such as detailed feedback reports of 
questionnaire responses to each prggramg were efforts to 
encourage use of the information by program participants. ^ 
Interviews during staff visits to aj^out half of the participating 
progr^EEia Indicated that programs were using the data, or were 
planniijg to use, it, in a variety of ways. However, the uses and 
the usefulness of the measures ca^ only be suggested in a very 
tentative way In this rerort, and need to be explored In 
considerably greater detail before this research question can 
be answered with any contldencep 

FrogrMi Asfe ssment: A Rationale for Multiple Indicators 

As indicated earlier in this chapter, the most conmon procedure 
for assessing quality^ln doctoral programs has been r^putational 
ratings. This procedur^relies on the Impressijns or opinions of a 
doctoral progrm that are held by faculty meters in other doctoral 
progrwns jsHthln the same discipline, or by other experts In the same 
field. The result Is a single, overall Index of quality* 

The multiple indicators approach to the assessment of quality, 
on the other hand, would normally Include the perceptloift and 
opinions of those within the program and also would contain 
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Anfanatlon about a variety of program characteristics (fee Table 1,1), 

TtkB ttultlple .Indicators procrdura makes it po&sible to exwine 

deputmaiital strengths or wetnesses in different progrui areas or 

featuirast md might well lead to the conclusion that, when a 

departBent Is conpired to other departments in the same discipline, 

3 

it is strong in mamm areas but weak in others. 

A major pronise of this study was that Judpients about^quality 

in doctoral programs will be better if they are based on multiple 
' t 

indicators of quality rather than on a single indicator* Wimt do we 
^mean whm we assert that multiple indicators are "better"? 
Essm^^ally we mean that^ when compared wich single indicator , 
nulcipla Indicator procedures have at least four distiiict advantages. 
They (1) are fairer, (2) are more useful , (3) reduce the problem of 
"halo effect i" and (4) are more likely to stimulate Internal 
thinking abou^ and discuyHlon uf program achievement a and shor 1 comings . 

A mu 1 1 id imen 8 ionaj^ pro c edu r e jL^ ^f a i r e r . The use of A single 
indicator of doctoral progrw quality is "virtually certain to be 
unfair to some doctoral programs. If, for example , the single 
indicator employed is jne that assiuaes a research and scholarship 
modfl for the doctoral progr^^ (as the quality of faculty ratings 
colleeted by the ACE clearly does, for example), then doctor^ 



3 - ' 
"Program" is used throughout -this report to refer to the 

doctorate^ level offerings of a parmcular academic unit within a 

university » Because this unit usually. Is a departments the tmtm 

tend to be used interchangeably. However, the assessment procedures 

reported In^ this study were not designed to evaluate nondoctorate 

functions of university departments and, therefore, are incomplete 

as ^yrall Mttures of departmental quality. 
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progrms with other purppses will be penalised. Considarj for 
lnsttfiea» e dOeioral ptogram in paychology that emphaMlEes the 
praparatlon of clinicians or a doctoral progrM In the himanities 
that emphui^as the preparation of college teachers. Neither of 
these programs wuld be fairly judged unless particular attention 
Were given to quality indicators appropriate to their pt*ogrM 
purposes. Though the research and scholarship model clearly is 
the aodel preferred and claimed by the great majority of doctoral 
programs, there are neverthele s a nu^er of exemptions to this 
tendency* And If the purpose of the evaluation is the review of 
individual programs, those programs th^t do OTphasiie the preparation 
of teachers and/or practitioners will ^ot be given a fair opportunity 
to demonstrate their strengths. i ' 

r 

■ , i ^ ^ 

A multidimensional procedure would be far mot^e useful . One of 
the glaring weaknesses of any unldlmenslonal procedure for the 
asfeasment of quality-^hether at the dppt^ral level or my other 
level of education — Is that the Information la seldom very helpful 

foT a better understanding of a specific/ program's strengths mfi 

I 

weaknesses. Participant feedback about ('various specific dimensions 

" 1 
of a doctoral progrw^ — such as the students' ratings of the quality 

of teachings the faculty ^ratings of the- adequacy of the library 

holdliigSp or the alimni opinions about Ithe dissertation experience'^* 

prpvldM the stepping stones for program improvement. ... 

Multidliasnsional procedures reduce^ the problem of "halo effect." 

One of the major shortcomings of peer re^utational ratings la that a 
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"halo effect" cl* 4rly serve^^ l*' the uiiw-trrdfiLcd ad^ntage o' cerLgin 
programa. Such b^lo effects occur in one or both of lwq ways. 
Flrstj^some weak departments benefit from the strong rcjutation 
of th%±T universities J anc thus receive better ratings than might 
be warranted simply because their a idemic location* Second , 
some departments may be rated on the basis of their strengths in 
batter days; they may be ra^^d not tor what they are but to^ wliat 
they wer%, (What migiit be called a "reverse halo" . m also occur 

we^ reputation of its institutic ; or by its own weaker statas of 
yesteryear,) A multidimensional asaessraeT^ procedure, especially 
if it leans heavily on the insights , experiences , and opinions of 
people in the program^ would obviously be less affected by the 
problem of halo* Furthermore , multidimensional procedures would 
tend to move thinking away from the bipolar notion of departments 
being "strong" or "weakj" and might refocuSj instead, on the fact^ 
that departments are usually a combination of some strong points and 
some weak points, and th^ frirV M^n^l fv i-n mmtnh the^e strengths 
and weaknesses to program purposes. 

Hu 1 1 i d i ni^n s iono-l ass c s s m :i t pr ocedures are raore likely to plac^ -' 
emph as 1 s on t h^e^j ro^ce s s . i we assume that the multidimensional 
approach will include a onsl durable amount of feedback information 
from students J faculty ^.embers , and alumni, then it--is clear that" 
program evaluation is les^something that happens to the program than 
something that paopie ^n the program do together. The process of 



self»evaluation almoBt rilwAVH carries w-C : the c^tcn-helpf ul 
■byproducts of thinking aDOL_, iiscu^Bing, ie^ iting, and compromising 
upon issues important to the future of a giVuin prDpranu Faculty 
nicmbars who have p:.rticipated in ins t i tu 1 1 onal self-study efforts 
often rep.-rt that the process of the self-^study was every bi: as 
important as any summative r^.^port. . ach pr ess can bring aDOuC 
a certain amount of internal ui oi.. >rL, i)0' ^ tor sr.ua«.:iiCa au^ 
facult^ . but the long-range conHequences U'r tlie iepartment are 

Examples of Multiple Indicator Frcfilos for Doctoral Progr#h^ 

In order to illustrate several possible approaches -o 

Interprating multiple indicator information about doctoral progrim 

quality, two contrived example profiles are presented in Figures 1.1 
% V 

were arbitrarily chosen from the 30 indicators listed in Table 1*1. 
(It is important to emphasise au Luib puluw= Lhat theSe /aidicatcra 
are not necessarily ones that emerged as particularly Important or 
reasonable from this research. They are listed in th^e figures as 
eKamples only.) 

In Figure 1.1 an example profile of a hypothetical chemistry 
department is sketched. Note that one attractive feature of this 
approach is that the department ^ua..ling with respect to the 
10 mjaltty IndiratDr^ is given in comnar^son with other d^toral 
p^grajns in chemistry. In this\ particular example, the chemistry 
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Figure 1.1, Examp°^e ^ Depa r t:""*en c ^rMff.]^ 

— 3 

Ten Possible Indicators Proiram Quality" 



Compared with othtt doctoral programs 
la chemistry j this depai tment isj^ 

On^ of the About One of the 

lowest ^^'^U^^ highest 



F aculty I, 

(1) Research productivity 

(2) Teachi^v^ ef tectivenass 
(jj Croup Tnoralc or ccpric 

Students : 

(4) Acadattflc ability at entrance 

(5) Accomplishments of graduates 

(6) Satisfaction with program 

Resourc es ■ 

(7) Financial suppdrt 

(8) Library 

4 

Operations i 

(9) Program leadership 

(10) Efficiency of degree production 



^This is an example only, constructed to be illustrative of the 
advmCages of a multlple^^ iicators evaluation model. The ten Indicators 
were chosen arbitrarily from Table 1*1 • 
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department is considerably abcve L-..c uwcz^c ot othar che"".istry 
dapartments with respaCt to research productivity and financial 
iupporti somewhat above the average ' other chemistr>^ doctc ;ai 
programs with reBpect to scudant academic ability and library 
facilitiaai about average in program leadership, but below average 
in tanas of the accomplishments of its graduates and current student 
satisfaction with che program^ still firtrer below the average 
other dapartments for efficiency ' degree production and teaching 

faculty morale or esprit. In effect, thl& make^believa chamistry 
dapartmen^j whan compared with other loctoral programs in oftemistrys 
appears to be faring quite well on the several traditional itidicato 
of doctoral program quality (e*g* , research productivity, studaat 

s 

ability's facilities) that are often fairly well-known about programs, 
but is not doing naarly as wall nn savaral of the less traditional 
and gan^rally lass wall-'kno\im quality indicator^ (e.g., faculty 
morale, teaching af factivaness , efficiency of dagraa production)* 

luggasted hara would accomplish two things: it would provide 
inrormation i^out more IndiuaLur^ uf program quality than might 
otherwise be considered » and it would provide all snch^lnforaation 
about a program in comparison with measures of the same character- 
istics at other doctoral programs in the Bamm discipline. 

At this point the careful reader may have a number of important 
questions about the mechanics of the sample profile ptesented In 
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Figure 1.1* For exafflp Ic* , is it p'"»s s i i.*' 1'!^ i * t: -i 1 n in i arma t i on thci l 
would be acceptable as measures of snme of the tf listed indicators 
of doctorpt program quality—eb^.^uially such characteristics as 
teaching af f ectlVenass , group morale ^ accomplishments of graduates, 
program leadership, and the likeV Arii thjugh the example profile 
indicates that one's own department is being compared with the data 
from "other doctoral programs," one might want to know what other 
doctoral programs? All of them? Only a chosen ^ew? These questions 

answering such questions, however, we ask the reader to postpone 
major concern about them, and to continue te e: plore with us the 
conc^^C and pf^tenti a1 of ^ mult Iple^indlcators approach to the 
assessment of doctoral program quality* 

We have used Figure 1.1 as an eKample of how a multipla-indicator 

_s 

approach might be used to comp^^re a given department * s standing on a 
ntmber of Important cha i^teristics with the standing of other 
departments in the same discipline. But what ibout the possibility 
rif n^fnri^T^ij^cf nr^^ r m&n f= rAl^»=ivp fo othpr Drograms in the same 

discipline with a progrF.m in another discipline within the sme 
Inatitu^luo? Such compaiiEons arc also' pOGsiblc , and one example 
of how Miey might look is sketched in Figuve 1.2. Tlils is essentially 
the Baoii as Figure 1.1* except that in ^igure 1.2 we have included 
contrived data for two departments ---psychology and soclology'--'within 
Che ' ame institution. The data connected by solid line© (the 
sociology department) suggest that this department's research 



Figure 1.2. Example of a Corparlscn cf the Profllt 
Two Depa^ menf^ Within the Same Institution 



* onpareQ with other doccoral prugrams in 
psychology and sociology, the psychology 
and sociology departments at this 
institution are* _ 



lowest 



high est 



Facul ty : 

(1) Research productivit% 

(2) Teaching effectiveness 

j t 

(3) Group moqale^or esprit 

Studants : 
€ — ^— 

(4) Atadamic ibillty at eaLiai'Ce 

(5) Accomplishments of graduates 

(6) Satisfaction ^th program 

Resourcea * 

(7) Finmclal support 

(8) Library 

Operations 
=^(9) ^'rogram leadership 
(10) Efficiency of degree production 

^ ^ ----S ociology 
^""Psychology 
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productivicv Ik domewha- helow the average f other sociology 
departments, but above he average of other sociology urograms on 
taaching effectiveness, trnur morale, and so on The 
raprasented by broken lines (the psychology departmant) indicate 
that this dapartment is near the bottom of the distribution of 
other 'D'"' nholDe*^ dt irtnisnts on smc^: ch?TactP*ri?*''^ ^rnti-n 
morale and program leade r^hip . buc lares better when it cjraes to 
student acadeniic auilityj accomplishments of t^raduates, and the 
like. Thus, when comparL^d with other departmants in their ovm 
ra^pective discipline this institution's sociology dapartmant 
saams to be- in boroewhaL betuer condition on thasa ten Indicators 
of prograsi quality than is the psychology departmant, (Nota that 
the coffiparlsons between departments ara made with refarance to 
dapartmants in the__ same discipline at othe^ univafsltias , A more 
aacailad digcussion or rnxs poinL j^nuj^ua^u aii jmmt^ wii^put^j.*/ 

As with the fict ^icus profile prasantad In Figure 1,1, there 
are savardl _qtiestions that one might want to raise about the spaclfic 
procedures and measures needed to put together the kind of data that 
are presented in Figure 1,2, At this point, however , it is the 
cuncept of a multiple indicators procedure that wa are tryl^ to 
develop. Assuming for the moment that reliable information can be 
obtained about a number of the more Important characteristics of 
doctoral program qualicy, the advantages of such a procedura for 
doctoral '^ro^rs^ ^^fni^^^^^^r' n^r^ OM>^ ^r^nf -f . j^efYimT reliable ' 
iafon^tion can be obtained for such a purpose is the basic question 
addressed In this report. L 
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C^e final point ab jt uch multiple indir.anory of program 
quality would seem tr: be ap rppriata^ Though there are some doctoral 
•progfam characterist s that virtually all observers would agree are 
desirable ones for any doct jral progra:^: \o nave^-overal! po-iLive 
faculty morale^ for examp.fei, ^r gi.i.^uaLdS wuo peiifurin wel.- in their 
suhReauenf c.are^rs^=1t "ipv ""rhelf^ss true that man"^'' indicators? of 

program quality at the doctoral level may be "good"' for some kinds 

/ 

of programs and ''bad" for '--rhers- lliough most ioctoral programs 
would hope for a high "score" on s^ch. characteristics as reaedrch 

productivity or student academic ^bility at entrance to the program * 

/ 

, / 

even these indicators wc^ld net ^ ccivc unifo-'s endorsement among 

/ , 

pragrams at the doctonl level. Thus ^ in effects there would be no 
such thing as an optimum profile appropriiite for all doctoral 
programs* Instead ^ the prof 'le for each doctoral program would 

Overview of ^jthe Research 

The procedures of SOTple selection^ questionnalra development i 
^RtB rnllert-tnnj and daf^ analysis are detailed In Chapter 2. 
Chapters 3 and 4 discuss the general Issues of program quality 
rAtlags and ways to charactcrisG different program purposes. 
Chapters 5 through 10 present research results on ways to measure 
pifogr characteristics in six areas i training and productivity of 
the faculty, academic ability and achievement of students * phyalcal 
and financial Resources » the academic and work environment, the 
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academic program^ and alumni i riiavemencs , Lnapter iJ suimnariier 
the resaarch rasultf ana preHont? ■ jrmative nrifiies on sti^ccej 
variables for each ot the thrGe discir-ines» Re-ults of the 
reSw^arch dre evaluated m tni- cnapcer, nn^' wayy in^l^^ch tb^^ 
may be usetul to the gradvuire commMnity ar^- suggested'* Readers 
may wisl to turn to thi- chapttr fi t, and the^ return to ine 
chapters that present the reheard . ^^ocedures and results In dfeta 



STUDY PFi:'"^EDURES 

The basic purpos.. ^ of the study and the r^ijor ques* ions that 
guided thr- d', igr nf t^ -- res- irrh are discussed in the previous - 
rh,nnrr»r . Thr^r ^r-rr hnw'-vr r . • r^"=-1v drc 1 F": ior.F concerning 

parameters of th.-- "study Lhat snwald be noted here* 

1, The study would be eKplurai. v in nature and, therefure^ 
limited to doctoral programs in three arts and sciences 
disciplines in 20 to 25 univers ^ti . The universitieB 
would be selectea lu represent: c"ne iull range of doctorate- 
producing universities in the United States, 
2* The research weuld assess many aspects of doctoral. programs , 

emphasizing the chriracteristics that were judRed most important 
to educational quaiity by graduate deans #na liated iii labi^ 1. 
Because of ^ widespread and continuing ^intftrest in the ACE 
reputatlonal ratings of programs, new p#er rating& of taculty 
quality and program attractiveness also would be obtained but 
only iwr the limited |number of programs in the study* 
3* If at all possible, one set of data collection instruments 

would be developed for use in all three dit^ciplines * in order 
to maximize the poaibllity of comparisons across as well as 
within disciplines, 

qualitative and quantitative aspects of programs, aggragating 

i 



questionnaire resp^mses or v.; ■:- : aininp infor .Ation from a ^^tt- 
mental records to deacribt^ the p' jgrams in w^ys that permic 
ccmparlsona between programs, Iab constr^ ution of profile" and 
scatistic. i araly^^ris. kl^huu^Ai -% . -rge number of varia^ l^b 
were to be assessed In lUi. ^i^d, , ^ ---^ refa^^arcL 

was rn identifv mAnaReable nuT^'ber ^-^f measur ^ that demmistrata 
satlsfaLi reiiabiiitv and illdirv as indicators of important 
program character sti ci^ . 
5. Wherever possible^ a; •=^,ntlon would be given to questionb con- 
cerning the feasibility and utility of the assessment procedures 
for future use, ir luding nethods and cos^s of data collection^ 
proct^durGs for tlic feedback of data to participating programs, 
and ways in which the information about individu ± programs 
might be used by nart icipat ing institutions. 

The proposed study was approved and funded by the National 
Science Foundation to the Council of Graduate Schools beginning 
July Ij 1974, Educational Testing Servlcej which had conducted the 
earlier survev of graduate school dedans for the Graduate Record 
Examinations Board and had developed the research proposal, con- 
tracted to conduct the ^eFearch The researchers ^ere guided and 
assisted throughout the project by a hard-working steering coofflaittee 
of graduate deans who were appointed by the Council and the GRE Board, 

The disciplines of uheniistry, history, and psychology were 
selncted lor study bt«^^' they represented different major areas of 
academic eiidaavur, had '-established doctoral progr^ns in must 



major universities, and had indicated _on-ern about cht problems 
of doctoral education through participation in earlier studies* 

tacL with an officier in the major professiuaal associitlon of 
each discipline indicated that tnere wouid "-t a nigh level of 
interest in t^ tliat w^i.. prupg^ed, and that . ^^i^ woulJ 

af least informal - nrnurafiemen t within t^^^e disci pllnp f ir proEram-^ 
to participate* 

An additirnal rnascn ^or ri^-idl:;^ to study thre^ diff^:^nt 
disciplines was to provide pra * Leal i^si of wliether '^r .ot it 
was feasible to use one set oi data collection instruments in all 
of the departments. The nbility to u^e the same ^ues tlornaires 
and forM in a number^ of disciplines had several practical 
consaquences* For ipxamplej if different queBtionnalrea had to be 
constructed for each di cipline, data collection would be much 

to cnmpare programs with'in that discipline. At the same time, 

8tructur#^^ and culture of the disclpMne^ In the study. 

To help^ us understand the major concerns in these three fielde 
and to Inquire about them as appropriately as possible In the 
questionnairps i a small advisory coimnittee of prominent acadafflclnns 
in each of the three fields was appointed by the project steering 
comittee. Each advisory cormlttee met with the researchers to 
critique draft versions of the data collection instruments, and 
alFo reviewed drafts of thl« report of the research results* 



Sample Selectio n and_EnjJ^^:^ 

The first step in s-xecting a ^arr^plu . : 2^^ tc j5 universities 
was to ideneify thoF ^hai granted doctoral degree^ in all three 

f ields--^chemistry , histor/ , .liid p3yi^:iuLo^ . A ,^n:pi :.tc iiq^ ::r 
these 114 .astiEuli^na will u IjunJ in AppGn.lix The list 

inciudes mosf of tru^ un i vers i r 1 e^^ that uroduce -^cctoratas ir these 
, lelds with Lhe inception of tc^chnical iastituCes in chemistry ."nd 
professio. ^ graT^s in psvchology. 

In order tc b ^are tnat the sar^ple would include a number of 
tiie departfne:. that ^ rain tnQ greatest number of students in eacu 
field, the list nni^^f^r^i f ir^s granting degrees in these three 
flelis was ordered by total number of doctorates awarded in a recent 
three year period. The list was then divided into tour sections so 
that each section represented the production of one quartBt^f\the 

^^^P^rs^^i-n^ fTT-^ni-^H Kv ^11 of the universities i the 
specif l|d time period. A random sample of Sevan Institutions was 
^^IttUL^U Ilmu. tht fir:. I two graupD and -i:^ ir-tic-tlon^ 

from each of the third and fourth groups for a total of 26 univer* 
sities in the btudy bamplfe (Mos^s i Oakford, 1963), 

Table 2.1 compares the sample 26 universities * \i the unlx^irse 
of 114 universities on public and private coi .ol, geographLc region, 
and renutational rating of the quality of the graduate faculty. The 
sampling procedure produced a good distribution of institution:! 
settings and program sizes. It also resulted in a fairly unii^^m 
distr5,bution of reputatlonal ratings aurnss the three disr^' 1j. j» 



:3D 

"I 



Table 2.j 

Cc-nparis m of Sampled Institutlcns with All Uni' >rsiciea that Grtint 
Oortoral neert-es in Chemiis^rv. ;;L.rv, and Psycnology 



ERIC 



N % 



Control 

-^ublic 7C. 67 73 

p^-s,,^^,-, 1- ''1 7 7 7 



Region 

Nort^ea^t il iu i 04 

East 23 20 4 15 

Midwest 29 25 9 35 

o U U L ! I ^ ^ ^ ^ = 

West 26 23 6 23 



Progi 



3.0 or above 32 28 11 42 

2.5 - 2.9 18 16 6 23 

2.0 - 2.4 14 12 3 12 



*Roose & Andersen, 19 70, 
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Psychology 

3.0 above 31 27 1* 54 

2.5 - 2,9 15 13 2 OS 

2.0 - 2,4 21 19 5 19 

Not listed 47 41 5 19 

Historv 

3*0 oi abova 15 22 11 42 

2*5 - 2.9 10 09 4 15 

2.0 - 2.4 21 18 5 19 

Not listed 58 51 6 23 



^ ^ 2.6 

with smp progr^^ at ameh level* There were a larger proportlod^Qf 
pragroM' iM the sraple with reputatianal ratings of 3.0 or abave 
a ra&llw proportion of prograraa In the "not listed" category beeause 
of the stratified ui^llng procedure. Thl? was, considered appropriate 
beeMse the study was particularly concerned with the assessment of 
program characteristics as8ocla|M with a high level of educational 
q^ilty« Each of the dlselpllne advisocy eomlttees felt that the 
rMuitlng list of universities provided a reasonably representative 
of progress In Its fields v 
The president of the Coimcll of Graduate Schools thra Invited 
the graduate dean at each of the 26 leleeted universities to 
cooperate In the study. Twenty-five agreed to participate and 
provided the nmmm of u cmpus coordinator to manage the data 
collection. Copies of ^he Invitational materials and guidelines 
for on^cmpus data collection cm be found In Appenilx A* 
Participating unlveyeltles are listed In Table ^.2. 

* 

QuestlonnAlre Development 

_ _. _ - - ^ ^ 

Information about each program was sought from five sources: 
faculty m^Aers who teach doctoral students, advanced graduate 
students » alunml three to five years after receiving their degrees » 
department chalnen 9 and departmental records. Topics covered In 
the questionnaires were guided by the list of progrw characteristics 
developed from the survey of graduatt deans and shown In Table l-li. 
and by the dems' ratings of a variety of possible pyogrw Indicators. 
Details of^ these ratings can be found In Appendlsc C* 



set on Dlmmsiofis of Quality In Graduate Education 
Participating Institutions 

Boston Collage 

University of California , Berkeley 
University .of California , Davis 
Uni's^rslty of California , Los Angeles 
University of Colorado ^ 
Emory University 

Florida State University (Tallrtassea) 
Indiana University (BloOTilngton) 
Univarsity of Iowa (Iowa City) 
University of Kansas 

Louisiana State University (Baton Rouga) 
^ University of Maryland 
University of Michigan; 
Univarsity of Minnesota c 
University of Mlssonrl (Columbia) 

New York University - 

* 

NorthwastMn University 

Okl^oma State Univarsity (Stillwater) 

University of PannsylVMiia 

Princeton University ^ 

Stinford Univarsity , • 

University of Toledo * . 

University of Utah - . 

West Virginia University 

University of WlsQonsin^ Madison 




T 4^ ' 
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Five kinds sf qucstlDna v. ire. Included In the student queatlon^ 
nslifti (1) Mtent of agreement with several statemants about the 
depactntnti (2L ratings of the dapartmant's faeulty, faellltles, 
n# eurrleultffi&l (3) cnaraetarlzmtlon of students In the departmanti 
badad on "crltiaal nbldents" of outstandingly go^od or poor studants 
as Idsntlfled by faculty men&ara (Rallly, 1974); (4) perceptions of 
progry raphasls on the preparation of rasaarehars, teaohers, or 
other praetltlonersi and (5) personal Information* Respondrats were 
tovltad to Ad eonmients about ^lieir department and ^reaotlons to' the 
questionnaire on the last page* Copies of all quaatlonhalres used 
In the study are Ineludad in App^^ndlx D« 

The alumil ^quastlonnalre waf designed to paraU^ the i udent 
questionnaire wherever appropriate^ In order to obtaf ^o dlffarant 



vl^Hnhts about the ssmm aspaets of graduate departments.. In addl* 

tlonf altsml were asked to evaluate thalr dlsaartatlon experlenda^i 

/ 

rat^ their satisfaction with the. program and, the extent to whlcU^ It 
prepared them for thalr praaent work, and report ppstdegree ^ 
scholarly accomplishments* 

The faculty quasflonnalre also paralleled the studMt question^ 
nalre In structure, although only the. set of Items to charactarlEa 
students In the dapartment and the Itims concerning^ parcieptlons of 
progra^ emphasis ,were exactly Identical* Other Itras ware selected 
to obtain 'a "brief but comprehensive view of the dapartment from the 
faculty p Aspect Iva* Faculty members were also asked to estimate 
the averaga weekly amount of time they spent on various professional 
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aetl^tiMi «d to report pairtlGlpatlon In protessional associatioiii> 
wuda» publleatlonSi preisntatloni, and background eharaetMlstici. 

Ifi addition p faeulty TOnbars ware asked to rate the quality of * 
tha graduaLa fafiulty and the attractlvenaas of tha doctoral program 
la thalr flalda at tHa 25 imlvarsitles In tha study. The smie 
fonat waa used for thesa ratings as that uaad in the Mariean 
Counell on Education reputatlonal ratings (CaTtters 1966 i Roose 4 
i^darsanp 197b). Tha results ara reported and compared with tha 
earllar findings la €haptar 4* 

The Items of information raquested on tha departmental proflla 
fora were based on Indicators rated important and avall^le by tha 
panel of graduate deans in the earlier study (see the "University/ 
Dapartflient. Retards*' section of AppendiK C). The Information Includes 
the nta^ar of students and faculty fl^anbers* adalsslons data, 
financial assistance data, faculty research Involvement p degrees 
granted, plaeement of graduates, and so forth. A cemon form to 
collect this kind of Information proved t© be the most difficult 
quaatignnaire to construct because there is little aireament mong 
^nlveraltlas on ways to define these characteristics or activities. 
For imich the eama reason* it was also thai most difficult queatloraalre 
for progrra rapresentatlvaa to complete* Some programaldld not have 
the records needed"^o provide the requested Inforroatloai other 
rapraaantatlvafi reported that compiling the Infornatlon In the form 
raqueated would be Inordinately tlma-consumlng and expmslve. Even 
when the Information was reported, It was often accompanied by a note 
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ef im^rc^fcy about Ita rellablllt^ For these ressons^ cmly 
seleetttd ItMa from this ^Nq^^^^e^een used In reporting and 
oalyslag reaulte of ^ the researeh« The Item that vere evaluated 

■arl^ on the Qopy of the qutstlonnalre in Appendix D* 
ProblMa i^tii 80M of these data are discussed ^n Chapter 7 as well 
as In othmT p^ces In the report. It Is Interesting to note hire* 
hoveveri that the InformatiVi that the graduate d^sns said was a6st 
readily > avaiisble on their empuses (e*g. p enrotlJient Md e^radlture 
data) tuned out to be the hardetf^to eolieet to ways that would 
be eo^iarable aaross progrmsp while the Judipents ud opinions of 
prograB participants that few departments collect with my regularity 
could be callaeted relatively inaxpenalvely , efficiently p and 
reliably. ' . , . \ 

The last questionnaire wiis a brief form for the departm^t^ 
chairman to^conpljte« It asked about some departmental praDtlces 
and experiences in recent years p requested ridings of other 
departments In the same field on the same form as In the faculty 
questionnaire « and inquired about the procedures used to conduct 
the on*-CtfBpus data collection for this study, 

^ Data Collection 

, A ca^us coordinator was appolhted by each graduate deu to 
mmage on*cra&pus data collection for^ the study. Faculty ud student' 
questloraalres Were shipped in bulk to each coordinator im i^rll of 
lf75p distributed and collected by departmmtSp and retmrned to 
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Edtt^tiMUl TMtitig Servlgs»by the coorditiatori Each questionnaire* 
AeMBpnlad m mvelppe which could be sealed to protect the ^ 
orafUMtlallty of the reply; only the respondent's nue and depart- 
, ^ ^MAt mre on the outside, to facilltata follov^up of nonrespondrats# 
Departanta were tostructed to distribute faculty quMtionnaires to 
all. faculty mmbers who teaeb doctoral stud^tf » and studrat ques- 
tionnaires to all advanced doctoral students (defined as usually ^ 
in their third fear or beyond of graduate study) . Guidelines for 
on*"caiipus i data collection will be found in Appendix A« 

Trtle 2,^3 swoarizes the reports from, the department ch^iraen 
, Aout phe data collection procedures* Distribution and collection 
of the questionnaires were most frequently handled by a departmental 
secretary p usually through^c^(pus mail. A departments limited 
' stuifent questlpnnaire distribution to students who had achieved 
candidacy status and ^therefore we^ not on campus with any regularity. 
Theae department^ and sb^ others mailed quescJoimairea to itudents« 
often at considerable trouble and expense^ -Some depar^ienta also 
mailed questionnaires to faculty meters, particularly if the 
qyeptiononaires were distributed whm regular classes were not In 
aeasion. 

About two-thirds of the deparoents reported that they followed 
f Qp'on nonreapondents, usually with a menorandum or a personal 

tmmbiAmt^ Most felt that ^e procedures that were used were at 
leaat fairly effective and that most students md faculty aefflbers 
were willing or at least neutral respondrnts. 

Caapua coord^ato^^s were requested to obtain liats of persons 
who recelvild doctorate degreea from each of the three departments 

--^'-^ ■ ■ 46 , - 
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Tab la 2,3 

tepAvnmtal Eepart of Ftoeedurai Ummd to Ob t aim 
Student and Fuulty QuMtiomaires 
iin paremtAges) 



Wmmltf Quaatloimalras Studmt QuMtigualVM 



Chwm 
(NX (22) 



Hist 
(23) 



(21) 



(22) 



fay 
(21) 



'ti ^ 

1» 1^ MOAiad the dlatrlbutioQ 
aad aoUMtim of quastim- 





9 


35 


24 


faoiltj aeiAar. 


0 


17 


10 


D«pt» iiaaifltadt 


32 


9 




SarerataiTf 


55 


30 




Stodaiit 


0 




0 


Sam's offica 


5 


0 


0 




0 


4 


5 



a. 



nara the quastlonnalras^ 
diawtbutad? 
CmKgM Mil 
U.S» sail 
Ihtough claaaas 
At a faMlty maatlng 
At a :^ating of studciits 
A^aUabla to ba pickad up 
Otter 

3. Did ymi do a follow-up on 
aofir aapondm t s 7 

Yaa ' 
^ No 

4. What follow-up proceduras did 
you uaa? 

Verbal remindar (Maatlng) 
Haaora&difli rtelndar 
Seadad kiuestioimaire 
feraona\ contact 
Other 



S« Effei^tlVfaaaa of .distribution 
aad coilaotiOQ proaadures 
?ery effaotiva 
leirly efface Iva 
Hot affectiva 

6, Generei raaponaa to tha 
qyeatioDnaira and rasaarch 
Inthuaiaatic 
Willing 
Hautral 
Ealuatnt 
' leaiatnt 



14 




19 


0 


13\ 


10 


32 


4) 


29 


« 


57/ 


33 


Q 




0 






'5 


0 


4 


5 



59 


/4 


90 


59 


57 


81 


I ( 


13 


5 


5 


43 


14 


|l 


0 


0 


9 




0 




17 


5 


00 


0 


S 


0 


0 


0 


•9 


9 


10 


0 


17 


5 


5 


22 


10 


27 


4 


0 


27 


0 

N 


0 


68 


87 


71 


73 


74 


62 


27 


13 


24 


23 


22 


33 


0 


17 


10 


0 


i7 


5 


41 


57 


48 


45 


43 


48 


5 


4 


10 


5 


4 


5 


36 


52 


38 


32 


39 


33 


3 


0 


0 


5 


0 


0 


55 


57 


33 


50 


43 




41 


43 


57 


41 


43 


57 


0 


0 


0 


5 


4 




5 


0 


0 


5 


4 


. 0 


45 


70 


57 


41 


52 


52 


18 


22 


29 


32 


,30 


33 


23 


4 


14 


18 


4 


14 


5 


0 


0 


0 


0 


0 
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In llTOt 1971, mi 1972 » md to mmnd thOT to the rMsarchars so that 
i1\f%n1 qMfltlanaalras, eould b€ mailed dlregtly to thase graduatss* 
Lists of alml nuas and addressas wera received from/ alaoat ayary 
daparMmti 1a a fav cagasi a departnant praferred to eoataot its 
aliiakii dlfactlyp wid a dupply of the (quastloim^lras ^nd ratuctl 
aavalopaa ware provided for this pu^ose. Probl^iai locating aluml 
and obtaining responses from them are discussed In greater detail 
iA Chaptar 10. 

t CfflBpus coordinator also dlsttlbutW the departmantal profile 
ud^dapfctment chalrmu questionnaires to appropriate respondents » 
assisted In collecting informatlw for the profile formss and sent 
us copies of descriptive literature about doctoral programs in the 
three dapartments, . It would not have been possible to carry out ^ 
,the study without the asslataijce of" these staff rombers at the 
participating universities. . ' 

Qyeatlonnalre Responses i 

It is diff Iculn to "compucc accurate rasponse rates for the^ 
stud^t iaid faculty quest lonnftlres brcause of the method used to 
dlstrUute them* Each department estimated thfc u<mibai^ of quastira^ 
nalras it would need; they Wra sent tKiP nisabes plu/? about one-* 
th^rd a%tra» to cover loss and* to use in faUowlr.g ip on nen* 
ras^nd^ts.^ But It was clear thafr wany ot' th^.^m nuodbara were 
'"vary raugh guaases. Therefore, the departmrat chalrmm quest limnalre 
asfcad for the nwber of studrnt and faculty quef tilonnaires that 
ware actually distributed , Mid thes*e reports, were used to astimata 
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tmmpQMm rates . Th# number of usabla questlonnalras that upre 
raeal^ed ia discipline la Reported In T^le 2.4i 

In generalt It appears that abaut two-thirds of the student 
qiMatlcm&alres were returned » and that about three-fourths of the 
f acult]f aeibers returned questionnaires. Both the nimber of returns 
aftd totervleirs during site visits suggest that the faeulty m^iibers 
of aoat departments were eanvassed fairly thoroughly i ud that their 
ag^egated responses represent departments reasonably aecurately» 
Thm represmtatlveness of the student respondents is. harder to gauge, 
since departments could Interpret the instruction to distribute 
questionnaires to advancad graduate students In different ways. The 
liedira number of student re^ipondents per department is about 27 In 
^eh discipline, but the range is from sis respondents (from two 
departments that reported the distribution of 9 and 13 questionnaires 
reapectively) to one department in eaeh diselpllne with more than * 
100 respondents p Since all data in tlie report are simmMl^ed by 
department before comparisons are madep different numbers of 
respondents, per program do not influence the statistical results. 
Individual program Indexes based on only a few responses may, however , 
be someithat less reliable than similar Indexes for other programs 
that are based on a broader response. 

^out 90% of the participating departments In each discipline 
returaed usable departmental profile foms and department d'^a^^^an 
questionnaires. As mentioned aarlierp there was a high omission 
rate on some of the^lndivldual itans on the departmental profile form. 
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Tmbls 2.4 
Raapanse Rates 





Chemistry 


History 


pBychology 


1 * 

Usabla lAdlvl4ual returns ^ 
Iftfflber of prograns with usable data 


791 
24 


893 
25 


967 
24 


^ Irage of raspondents per ptogtsmi 

Lew 

Median 

^ ^ High 


6 

27 
104 


9 
29 
124 


6 

27 - 
202 


kVBtmgm rMponsa rate 
" Median respe^nse rate ' 
Minliirai respoQsa rata 


69% 
64Z 

39%' 


66% 
71% 
33% 


69% 
67% 
34% 


faculty ^qyestlonnalre 

Usable Individual returns ^ 
Ni^ber of pregrams with usable data 


511 

24 


584 

25 


598 
24 


Range of respondents per program i 

Low 

Median 
. ^ High 


8 

21 
38 


10 
23 
38 


9 
20 
61 


Average response rate 
Median response rate 
^ Mlalniini response rate 


80% 
•85% 
46% 


78% 
71% 
48% 


72%^ 

74% 

38% 


JluMil questionnaire 

Usable indlvlduBl returns ^ 
NualNfer of prograns with usable data 


430 
20 


349 
18 


393 • 
17 


'Ruge' of respondents per programs 

Low 
Median 
High 


10 
20 
68 


5 
17 
*3 


6 

19 

' 37 


Average response rate 
Medln response rate 
lUalauB response rate 


46% 
47% 

26% 


46% 

46% 
29% . 


45% 
43% 

33% 


Depsf tTlimtil profile* number returned 


2X 


23 


21 


Dept ■ ^alnun auestiLonnaire s 
Nt^er rAtuned 


22 


23 


21 



*tea daparttKat of ehmistry and one department of payehology 
deellaed to p^leipate in the study. Sufficlmt ua^le data were 
reetlved tmm ^1 pwftleipating prograns to include thra in the malyses 
of faeulty md stiidmt respona^.. 

Sm Chapter 10 for disoussion of alumi responses. 



« A datalled desarlptlon of the programi In ^hti study and the 

IP 

ehanctMlsti^a of respondents can be found In Appendix E. In 
gtasralp Che average profile across prograpu In each of the three 
dlsciplijieg Is quite similar with about 27 or 28 faculty mrabers, 
100 doctoral students » ^d 1^^ doctor's degrees awarded per year. 

, ♦ AlisDst all of the faculty regpondents held the Ph*D. degree 
(«a average of 99% in chemistry and psychology, 961 in histoty) 
and 98% were full-time mei^ars of the faculty Three ItCTS in 
coQimDn on the fitculty questionnaire and the departmental profile 

i T 

i 

provide one check on the validity of the information provided by 
these different sources; responses to these items are Reported in 
Table 2.5« The similarity of these rasponseb suggests that the 
faculty respondents are very much like all faculty members in their 
departments, at least inCthe reported ways. ^ 

The descriptive profile of studenti in Appendix 1 indicates 
that student respnndentB In the average prograra in each discipline 
had been enrolled in the program for four y^ars or mote. Uout 
70% of the respondents expected to receive the Ph*D* degree in 1975 
-or 1976, Most had done graduate study only at the university beln^ 
described and were enroll' a full time* In history said psychology i 
students in more than 80% of the progrms reported average gradua^te 
grade*point averages of 3»5 or higher | students in only 55% of the 
dmlstry progrms' reported average graduate «chool grades of 3,5 or 
above. Most of the' students were under 30, U«S, citizens » Caucasian 
and male. 
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Table 2.5 



Cofl^ftrlgon of Average Data from Departmental Records with 

\ 

Avarage Aggregated Data from Faculty Questionnaire MaponBea 

(in percentages ) 



Tenured appointment 
Froflle 

Questionnaires 



Chemistry History Psychology 



85 
81 



87 
78 



74 
71 



Outside reaearch grant 
Profile 
Questionnaires 



68 
57 



10 



37 
43 



Percent of facuity with 
highest degree from "leading 
institutions" 

Profile 

Que'atlonnalres 



84 
87 



91 
86 



72 
73 



V 



82 
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AlMst all of the data analysas have bem carried out se^arattiy 
for ,eaah dlaGipllne. Five analytical techniquas pradomjjiataiv 
!• IttiB regponse f raquanelae , Since a primary purpoae of tha 
raaaafch was to provide measures of progr^ia that would be 
usafui in self-studies and for progrm renewal , conslderfble 
attention has been given to the ways In which participants In 
progruis responded to Individual items on the questionnaires^ 
The Item response frequenclea for the student , faculty, and \ 
altmnl questionnaires were sinraariied for euch participating/ 
program In detailed feedback reports , together with comparison 
data for larf e and small departments (discussed below) * Three 
copies of each feedback report, were sent to .each university^ 
San^les of these feedback reports' and accoi^aaylng Interpretive 
materials will be found in Appendix G* 

In addition to the feedback reports to Individual programs , 
selected item response frequencies have been used ta the other 
^analyses described below a program's percent of faculty 

with an outside research gra^t^ or the percent of aluotnl 
reapOT/dents frpm a particular program who report "ejctr^aely 
good" career preparation)* Wlthln-progrffli! rMponses to items 
on the quasi-continuous scales of agrea-dlsagre^ and excellent- 
poor, as well as genuinely continuous variables such as *nu^er 
of publications or years of teaching eKperiencep were averaged 
^ and the mman scores used to represent programs on the relevant 

charactarlstics . 

■ . -53 
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Wlthto^dlsclpllne eomparlscn groups . Program response 
f^equmcles vara avaraged for subsets of programs grouped 
aceardt^ -to size or reputatlonal rating* Two romp rlson 
groups baaad on program si^e were provided with each £• icU 
tmpOTtt the avaraga response frequencies for 12 programs in 
the Sana discipline awarding tfie largest number of doctor' b 
dagraes In a lecent three^year period * and the averagr t espi 
fraquanclas for the 12 or 13 programs awarding a sti J b i 
nusbar of degrees # Program representatlveB could compare 
tabulatlona of respgnses by participants la their program wIl 
the average responses from programs of roughly similar size 
and could also sea if thcr* oppaared to be very diffrrrnr 
patterns of response for larger and smaller departments 
In their discipline. 

A similar procedure was used to construct comparison 
groups composed of departments with graduate faculties rated 
"d^stingulshad*' or "strong," "good" or -'adequate," and 
"iarglnai'- or "not sufficient for doctoral training'* by peers 
irtio partlclpatad in the , study (see item 25 on the faculty 
quastlonnmif a and Chapter 4 for details of this variable) . 
;for convenience^ 'these groups have been termed hi ;h, 
MdluBt ud lav reputatlonal groupings, and have been 
usad In savarsl chapters of this report to exarlne the way 
la irtilch a prpgram indicator operates similarly 
dlffarnCly In departments with varying reputations, both 
vl^^ and across disciplines* For the student and faculty 



in chemistry an ^ seve:= -■r.^nx-.^ \: ni^:.:v7 anri psychology, 
all with peer ratings of tb^ qL.alitv ^ the ' tdur^te faculty 
abnve 3.6 (i ^ distinguished , 4 ^ -trong, 3 ^ good). The 
low group ^onsist^ of -Ak programs in hemistry and history 
and five in ^sychologv. all wit! peer reputatlonaT ratings 
below 1.9 (2 - adequate^ I ^ not snfficient for doctoral 

rrs^-n.n^V Th- m-irlr^lr rrnnn in rinr!r un cf 12 procr; "^B in 

ja'-h di-.cipline ^itli reputat ior\ al ratings b tween and 2,1, 

Reputational rat in^.s were st lected as a grouping variable 
for these descriptive statistics because of a widespread tntara 
in the wavs in whicii other progiam characteristics relate to 
peer judgments, and because Lnese groupings seemed tu provide 
a useful way to consider tne distribution of certain other 
program descriptors. However, the. use of reputational rat\ngs 
as a grouping variable shoulc not 'be given undue weight in the 
interpretation ot the research results. i nare is variation 
among programs within these Fubgrouplngs as well as across 
all programs within each discipline* 

Scyle construction » Data analysis indicated that some of the 
items in the questionnaires were measuring similar aspectp 
of a program and that the responses to these items could 
be grouped to form a single scale score to rapreaent 
that characteristic, Comblnini items to form s^^ala scores 
reduces the number of Individual itens that need to be 
reviewcdp and alGO provides a prograTr.-levcl Indlr^tar that 
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tends CO be ever more coriSiste^^ and reli.^ Le z\ian BUinrnarifeH 
of individual iron resr-HF^-^^ A total of 'ii. ^c-At wc 
formfid frDm the ident que-.tionna^ ^ six ilefe fr^ji :.he 
faculty questionnaire, and tw^ scales troT^ the al imn^ question^ 
nair^s, Hie items in eac^ scale, Item^item and item-^sc^le 
cc^rrelations J and es::^rnatas scale reliability and homn- 

G.'y r^re^^nte.; in Appendix An analysis of viriance 

was coinpated for each scale to ermine that variation between 
pro;: ramB wa?^ citcr tbnn the within-program variance of 
responses ; only scales with significant were retained 

f^r use. T comp'^ ^ ^ Ion and vnr erpretri cion of each scale 
will DC diBCusseu aL an a| im jpi ia ^ u puluL the re^uiUS 
the Htudv are renwrted. 

Correlat j ons . Pe. *"son product^momant corrt^latiou coefficients 
of association between two variaDies are frequently used to 
present results in tills report. All correlations are computed 
within diBrin.ine.' us' Droerains as the level of analysis 



(e,g,, program mean b. lea or percent responses)* The nuisber 

ca^es email (ii^nal 1^ 74 nr IprsK but the aggregated 
variables are much more reliable than would be the case with 
Indivicual .-leanurenients , ' r we were us^ng in /ividual 
rather than aggiegated raLasure&, correlations based on 
24 or 25 cases (the number of departments in any one 
discipline) should achieve a magnitude of roughly ,35 to .40 
before being considered ..ignif icantly dlffevent from zero 
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(KcNemar j 1965)." Obviously triese wnuld :^onse ^ctve 
estimates of significance when the measures represt t aggregated 
dataj out there is no way to ndjust staList ically for this 
increased ^Reliability. ThGreforej since this repor^ is 
concerned with patterns of association between program char- 
acteristics rather than with differences in the si^es of the 
correl acions s the correlat ms will be fnterpr^ted as estimates 
of level of relationship* ar^ statistical significance is 
not reported. 

5. Scatterplots . One way to evaluate the meaning of a corralation 
between two varlabJes is to present a scatterplot of the 
programs in each discipline on buth dimensions. This technique 
is used at several points in the following chapters* A par-- 
ticular advantage of scatterplota is that the range of scores 
across departments on a particular variable is Imediately 
apparent s as are unusual deviations on one or both variables 
bv a few denar tnients * -itterolots also tend to remind the 
reader thai correlations represent geiieral tendencies for a 
^rnnn nf nrf-^gramH " rhough thev mav be hleh* there Is amnle 
Opportunity for meaningful deviations by Individual programs. 

^Confidence bands are also difficult to estimate for correla- 
tions based on a small number of cases because they aie not uniform 
over thr distribution. As a benchmark axiraple^ however j the 95% 
level of confidence r ;r a correlation of -70 based on 24 cases would 
range from *41 to .8b If the test is two-tailed, -^r between .47 and 
.84 for a one-tailed test (McNemar, 1965, p 138),. 



Site Visits 

After all of the data wore collected Anc collated and feadback 
reports had been sent to "he participating de^^artmenLS , members of tVie 
research teaffl visited about half of the universities in the sample^ 
These site visits intended to sain flrs^^hand information from 

the dapartments about t '^eir programs, answer questions abou^ tne 
departmental feedback reports and the study in genet dl, and attempt 
to detemine some of the ways in which departments and universities 
might plan to use the information that had een collected* A copy 
of the letter to graduate deans cgncerning s ' te visits 3^1 a guide 
for Interviews can be found in Appendix A, 

_ - i tlGUgn tnC £il^C Visits LIIC lU '2 ^J''2 ' f ^r' I? K ' ? i ^ * . ! i ^ ^ ? * m j 1 L » * ex i * ■ iujLJt 

of department chairmen and graduate deans* it soon became apparent 
that they needed more time to evaluate the us^ fulness of the program 
Informatioti they had received, '^hc vioits were helpful, howevp^, in 
sensitizing the researchers to some of the important campus l^i^s 
and suggesting interpretations for some of the research results* 

Fresentation of Research Results 

This technical report of the findings of the study is organized • 
around seven major dimensions of program quality and three progrm 
variables that appear to be related to several aspects of quality* 
The seven dlmeaas Ions of quality are (1) student bllitles and 
achievements (2) faculty training and performance, (3) physical 
md financial resources ^ (4) the environment ^ (5) curr-f lulum and 
aQad^lc procedures » (6) alusmi accomplishments and Judgments 

/" 
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about the progr^, and (7- overall jud^fments a' out a prograni's 
excellence. The last dimeriSion, verall judgTTients, is discassed 
in Chap tar 3* The others are discussed in or^er in Chapters 5 
thru ugh 10. 

Variables related several aspects of quality Include 
disciplint s program purpose j an i pragram size.'^ Only prograiti purpose 
is diacuafeed at length as an independent variable (Chapter 3) . 
Separate data for each of the disciplines are presented throughout 
the report, and program size is included wherever appropriate* 
A suimBary will be found at the end of tnost chapters. Some readers 
may prefer to read these surmnaries first ^ before examining the 

The final chapter also sumnarizeL the research results and 
th^n presents three or f jur indicators from each of the six program 
areas in thp form of a mul r Idimenslnnal prnflle whlrh can be ui^pd 
to compare individual programs with other programs in the same 
disciplj le* Measurement reliability, validity, ahd discrlrnination 
between programs were considered in selecting the Indicators to be 
Incliided in the profile* This chapter also evaluates the overall 
resu|.ts of the research and suggests ways to utilize the results. 
The reader with llmlt^^ ti^^ , or a desire to obtain an^-*lT!!mediate 
overview of the study ^ may wish to turn directly to Qu -r 11, 
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PROGRAM PURPOSE > 

The Ph. Dp degree was expected to by thu end product ol most 
the progrms in this study, thoug> programs offering D.A. or Ed*iJ* 
degrees were n^t ruled out. Thig^^proved to be .he. case; a Ph.D* 
was the reported degree goal of virtually all (96% to 100%) of the 
student respondents in each of the participating prograins. Since 
this is the traditional degree tor the preparation of researchers 
and scholars m the academic discipl - ugs , a high le\^el of eirphasis 
on the preparation of ' researchers w^s expected amoiir ^hese prograiiLS, 
tnerG arc at icn^t two ocner cas^Sj ri- 'ev'^fj iiihl ^^^t^t*^ (f^A^^t-^ i 
be emphasized as well: the preparation of college ter:hers, ^d thf 
preparation of other practitione s such ' clinical psychologists, 
museum curators^ or induatrinl chemists. Some variation of purpose 
was expected because recently a panel of graduate school deans 
ranked the preparation of college teachers as the most importmt 
task of Ph,D. programs in the humanities * and the deans repertad 
cbat the preparation of researchers and teachers received equal 
weight In the social sciences. Only in the physical sciences did 
moat universities clearly place iTimary emphiijls on the preparation 
of researchers. Though not as frequently emphasised as research or 
ttirhlngt the training of practitioners was ranked most importmt 
in each area at a few universities (Clark, 1973; 1976). 
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The same pane 3 of graduate s chool de ms C(^clu>^ed that there 
are no largi differences in the information needed to make judgments 
about the quality of doctoral programs designed to train teachers 
ot practit mers Instead of researchers ^ but there are differences 
f of emphasis chat make it important to take program purposes Into 
account when evaluati^ig their quality. For example, the deans 
assigned high importance to teaching skill in all programs > but gave 
It ftpacial weight in programs designed to train teachers # Similarly, 
faculty research skill and productivity were considered important 
In all doctoral programs but especially in programs to train 
i^esearchers , Training other kinds of practitioners may require 
mtemsnipo or a aegree or community invc ivement that are not 
nacessary in progrms with other primary purposi^s. The purpoees 
of a program have some bearing on the resources that arc needed 

and the ways in which these resources are allocated. And, of course^ 

1 _ ^ _ _ ' 

expect^ions for alunmi careers woald differ dspanding upon a 

progrm's emphasis. The recognitAon of variations In program 

emphasis also tepresents a source of pressure for thcj de^plopment 

of ways to assess many different aspects of graduate programs ^ since 

multiple program purposes call for multiple crlrerla of quality 

jet, t Panel on Altarnate Apprwaches to Graduat,^ Education, 1973; 

National Board on Graduate Education » 1973 and 1975; Education 

Commission of tne States, 1973), ^ 

One of the goals of uhis study was to consider assessments of 

quality in relation to program purposes* The emphases of pirograinB 
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ware investlg? ted by asking f acul t y ner ber - and ad van ced graduat e 
students to rata the degree of iT.portance their department currantly 
assignad to each of three different purpose? or functions: prc^parlng 
scht: ^ars/researchers s preparing coilege teachers, and preparing other 
practitioners. Each purpose was rated on a five^polnt scale from 
"axtraina importance*' to "no Importance. " Faculty meFbers also were 
aaked to rata the degree of importance they thought the departmant 
ought to assign to each purpose j and students were asked to indicate 
the level of Importance of each purpose t ^ thtir personal goals (see 
quas tionnalras j Appendix D, for exact wording and format)^ When 
surabed within programs, these data pro^^ided three different views 
It aacfi program's academic en.phasis- (ij the percent of faculty and 
^tudetits who perceived a lot of emphasis (responses 4 or 5 on the 
f ive '^point scale) on each purpose ^ and the extent of agraemant 
bfelnuea tlm twu aet^ uL obBaivera; (2) th^ digcrtpaiicy between what 
18 ana what oughu to ba emphasized, as a maasure of satisf action 
with the currjint emphasis and the preferred direction of moveoent 
In the event of any change; and (3) the avaraga faculty parception 
of the <; Ight assiCTt^d fo ach program purpose, ^e last maasura 
was con^ld^red tha bast Single indicator of current program emphasis^ 
lt>ai*a result a art»i prtf^t lut^d ard disc us sad in this chapter. 

Fmculty and ^ftudant P fcrc ep l i on ? of tophasis 

Th# average percenf: ov faculty members who think their programs 
currently plac^^: "extreme'' >r "considerable" importance on each of 
the thr^K^ purposes Is shov^ in Figure 3.1. • In this figure the 
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Faculcy-^Rated Program Purposes, by Discip^iXi^ 
Peer Rating 



and 



- Consii^derable Importance 



Faculty rating of 
eurrOTt progrm 

Fraparing schOiars/ 
rMaarehers, 
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practitioners* 
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p*^ )grams have been g^^ourv^d according to uhe peer rat ngs of their 
graduate faculties, as described in Chapter 2. Faculties of 
prograsis in the "high** groans were rated "strong*' or "distinguished*' 
by ^heir pears, clearly Identifying rh-^n, it tne mo^t prestigious 
*)rograms in the sample. Faculties of programs in the "Ijw" group 
were rated barely "adequate'' or less than adequate by their pee s * 
claar3.y distinguishing them as the least prestigious in the samr^e. 
The middle (and largest) groups uf programs are^ more variec but 
all are considered by peers to be better than "adequate" U their 
capacity to provide doccoral training* 

It is not surprising that the high prestige programs all place 

and much lesj emphasis on the other two purposes, as indicated in 
Figure 3,1. Thes are programe with outstanding research faculties 
end scholarly reputations; programs in the humanities and social 
sciences as well as in the natural sciences are expected to emphasize 
research. However ^ the continuing high level of emphasis on 
preparing researchers among programs In the n.edliMft and low reputa- 
tional groupings was less expected s particularly whan accompmled 
by relatively low levels of importance assigned to the preparation 
of collage teachers or other practitioners. On the basis of the 
reaults dascrlbed aarlier, it was expected that several programs 
In both history and psychology would emphasize the goal of preparing 
collage teachers, and that a few programs in each field would 
amphaaiEe the preparation of pracTlt loners . Insteadj only the 
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low^'rated progr^s in history assigned more weight to the prapgiratijn 
of teachers than researchers, mc only the low-rated psychology 
dapartments assigned more weight to the p.epar-^cion of practitioners 
than researchers* 

Though It Is impossible to know for certain why these data do 
lot agree with the perceptions graduate deans that programs in the 
hifflianities and social sciences place as much or more emphasis on the 
preparation of teachers as researchers (reported In Chapter 1) , 
several possiDite expiatiaL Luub w jii ue .:^ug^eetcwi* j.4Ast.*AAiww , 

could be cba. history and psychology are not typical disciplines in 
theAe areas of knowledge, althcugh traditionally large numbers of 
Tifsw ph,n.^<3 in bo^h fields bncome college teachers. AlsUi more 
emphasis on teaching is reported by the history departments than by 
chemistry or psychology departments, a fact that Is consistent with 
m expectation for more attention to teaching In the humanities, 

A more likely explmatlon is that the dems" assessments of 
program tasks reiiected the kinds of postdegree jobs that graduates 
tend to take rather than the training they received. In contrast, 
these faculty members may be reporting a lack of specific tf alnlng 
for t^ching or other professional practice in most of their 
programs, rhe lack of parMcular training for teaching Is supported 
by i t' i\mt responses to the statement, "This department makes a 
conslsVent efforr to help students become effective teachers," ^ 
average of only abOLit^30% of the students in programs In my 
discipline and any reputational grouping responded affirmatively. 



The only exception ^ia.s low-p .-=Hrig; pr jgrams in his tor , wheri an 



A tii-. possib] ^ expian Ion is t hat, since these are Ph.D. 
progrms y most fawjlty members could not bring themselves to 
denigrate" the importance of prep£.- ing schalarly researchars even 
when the prpgr^ in fact glvef only lip service to this goal, Thi^ 
possibility Is supported by ff;culty viewf: of the importance their 
departments ou^ht to asslm each prograra purpose as eported in 

preparing scholars /research^- r^ sii^. ild be ^ _J4itinued In high-pres t ^ge 
pi. i^'rama and should be h ighe r than at present in nedlum'- and iow^ 
prestige programs. At the same time, facu'*^/^ members in all 
prograny^ In all fields think their progrms should place more 
emphMls on the preparation ot teachers and also on the preparation 
of other practitioners* (Only low-'prestige psychology progrms are 
BXL exception) probably reflecting a desire for. slightly less emn^asls 
on the preparation of clinicians,) The daslrablllty of more 
atteQtlon to the prepmratlon of teachers Is particularly prevalent 
In both hlsto^ and ^yc^ology. Soma historians also thi^k they 
should begin to offer some practical training f^ rears other thm 



The differences between the fields «^re in line with the 
aetivltiea of the alitfnii of the fields* That Is, chralsts enter 



a or^ge of 64^* of the atuQ^ii' < .^gtt^u* 



teadiing* 



industry as often as they enter academe; historians are mostly 



employed in colleges aqd universities; and sizable nui^ers of the 
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aluml fff psychology pxjgranis are emplcjved as clinical or industrial 
psychologists p Alu^.i responses reported ir Chapter 10 confirrn that 
the purposes faculty think shQuld be enp' "'sized are mDre in line 
with the emplovTnent histories of . ecent pr< , l I graduates than ar-^ 
the parcalvad current program etpphases. The d' ' a do not answer the 
question of why mora .ittan^rion to carear preparation na^ not been 
forthcoming if faculty feel it ought to be part of their programs, 
but the indications that au mi changes are perceived to be desirable 
uy A iiuiuu^i ui i^uuiLy i ieuiu^^i r-' in. - eLiuQurage Bi iortb in Lnis 
directicn. 

Sepatate data are not presented .<r the student ratings of 
wUtrent prOEram emphases because t \ev are verv slinilar to the 
faculty results* In fact, students see slightly stronger emphasis 



levels, and slightly less emphasis on the preparation of teachers 
and practitioners than is reports ! by tha faculty. In tciTO of 
their personal goals , the student re&ponBes indicate that they 
endcnfse the high level of importance they perceive to be assigned 
to the prepar tion of reBearchers, but at the same time they would 
like to have much more attention given to the preparation of 
teachers j especially in history. Studenrs in chemistry and history 
are only modarataly interested in more emphasis on the preparation 
of other practitioners: however, psychology students at all levels 
-.^ould like more practitioner irientation, 
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The itrrngth of CTphasls on research tends to decrease with the 
decline in prestige of a prograa, though seldon wore than to share * 
Lm^uLiMmi^ -itK wne of the other progrm purposei. This m»y reflect 
the Interests of a less reaeareh-orlented faculty In the lower-rated 
programs, mA also probably reflects soBa dlfferancas Ija the 
specializations that are offered. The low-prestige progrMW toad 
to be small and to concentrate on one or Wo speclalliatlons rather 
^han the, full range offered by the larger and hl^-prestlge progrnis. 
They also tend to emphaskze the area of each dlsclplln© that Is m^mt 
relevant to applied cheers i organic chemistry (Industrial 
applications) , American history (public school as wall as coUege 
teaching), and clinical psychology. Average student specl.illzatlbM 
in the high- and low-reputationel groupings show the following 
contrasts s 31% of the students In high-rated programs and 60% of 
the students in luw-rated programs are in organic chemistry; similar 
percentages for American history are 311 and 51% I for clinical 
psychology, the figures are 27% and 46%. 

In summary, both faculty mMbers and students in almost all 
programs In all three disciplines perceive that their pregrMW do 
and should put major emphasis on the preparation of scholars/ 
researchers. Some prograns at every reputatlonal level also give 
some attention to preparing students for careers as teachers «id/or 
other practitioners. Both faculty and students thi^ this attention 
to career preparation should be Increased considerably, though not 
with any diminution of attention to the preparation of researcheis. 
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Om vlcir of congrumce of respmsti to quaitionf udna«ralng 
funnt pTQgtwm mmphmmim and desired aDphuis im provided In thm 
prcf^ous tAeclm. In addition, response dlscrepanoles of Individual 
respondents vere eonputed wd sumraarlied for eaeh dapsrtiBenip . ^tual 
md Ideal^ responses tha^ vtre Identical or that differed by only 
position on fhe five-point scale were considered to be In reasonably 
guod agrea&ent mad to reflect general satisfaction with the aurrent 
es^haslD of the progrim. Dlffetences of two positions or more on 
the-^seaie were considered to reflect diisatisfactlon anu :alrly 
strong desire for more or 1mA emphasis on the partleular training 
purpose « 

By the standard outlined In the ^bove paragraph » both atadents 
nd faculty were wall satisfied with their programs- emphasis on 
training scholars/researchers* Only in low-pre*"tlge progruis was 
there any marked dlasatisf action^ here, average of 8% to 17% of 
the Respondents wanted more emphasis on research. 

Studsnts, particularly p expresied dlssatif factlOT) Vlth the 
^^hasla currently given by their programs to the preparation of 
teachers. This dissatisfaction was especially apparrat in programs 
of high md mdltm prestige p where an average of 2S% to 37% of the 



studmts reported . that their personal goal In this area was not 
being met by their program (i«^.p there was a discrepancy of two 
positions or nore between their assessment of current eoqihMls and 
the aq»hi»sis they personally would like) . Faculty also saw some 
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nsed te inoi^Msa the sttantlon given to the preparation of temehers, 
putlml^ly in psy^alogy (15% in each re^utational proup) . 

isk «veraga of Aout^aOI to 25% of the s^ud^ts In progriuM in 
•11 gr^^s md* disciplines Indicated a fairly strong desire for 
nore attntlon to the preparation of other practitioners. They were 
Jolnrnd by diout a quarter of the facuity in moit history departeents 
«d by smaller proportions of the faculty in the other two fields « 

. In general I students expressed less satisfactlon-'^oore 
dlserepauey — faculty niei&ers* Only occasional menfeers of 
either group advocated less emphasis on any current program goal; 
If changes were desired * both groups were for more of everythliig* 

• •» 
Coffipatlbility of Diffyent Emphases 

Th^ third way of looking at program purposes was to average the 

faculty or sturfent resporse scale scores that were reported for each 

program '(5*e3ftrfeme Importanne to l^n© importance) and to use^ the 

average as an i^e^ to represent the program's level of e^hasls 

on a given purposa. Because the faculty and student responses were 

very similar, and ^fccause faculties traditionally determine a 

program's emphasea, the average of the faculty responses was adopted 

as the index of program purpose for use througKout the remainder of 

this reTport. 

Table 3»1 prese t^. mean scores ^ standard deviations, and 
tntetc i rrelntlons for average faculty ratings of ench program's 
emph usls on preparing sch jls^s/researchers , college teachers * 
ana. other prac vltlonera. The bar graphs earlier in this chapter 
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Table 3.1 

latereotMlatlon of |aculty Perc«ptlons o£ the 
Purposes of Their Departnenti 



Faeulty pareeptlons of 
^Bphasls on; 



1. PBeparlng scholars/ 
ressarchsrs 



Chemistry 

History 

Fsychology 



Correlations 
1 2. 3 



Mean 



a 



4.06 
4.18 
4.31 



S.D. 



.40 

.40 
.55 



2, Preparing college 
teaehers 

Chemistry 39 
Misto ' -50 
Psychology 08 



3.12'' 

3.65 

3.13 



.28 

.23 
.29 



3 4 Fraparlng othar 
pMctltlaners 



Chemistry 
-Hiatory 

Psychology 



■29 
-51 
-76 



03 
46 
-12 



3.13 
2.22 
3.26 



.21 
.49 
.81 



*5iiExtreme importance, 4^Conslderable Importance, 3~Soine 
Importance, 2"Llttle Importance, l-No Importmce 
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presented only the percentage of faaulty flt^sbers who dieeked the 

hljfli «d of the MSponse scale (options 4 ?nd 5) ; thou^ the aveimge 

gives equal weliht to all reajonsesp the raiphasls on. the preparation 

of sAolws/reaearehers continues to be paro^mt in each of the 

disciplines » with u average score well above "conslder^le*" The 

average ^^hasls on preparing teachers md practitioners Is lover 

(**8dBe*') I d at about the same level In chemistry and psychology. 

History came closest to equaling the preparatlm of scholars with 

the preparation of teachers (4tl8 vs. 3.65), while scorl^ lowest 

on the preparation of other practitioners. Heu scores for students 

are very similar » though slightly hl^er for ^^hasls on research 

in all disciplines md lower on the other two ^phases. 

The pattern of correlations betweM^the indexes suggests that 

the preparation of teachers mamt sraethlng different for histories 

than for chemi ts or psyehdlogiats. Accotttlng/to these datSi in 

chemistry and psychology the training of teachers does not necessarily 

go along with the training of researchers, but the two tasks are not 

percaived to be in conflict. Preparing practitlonirs , on the ether 

hmiAf was negatively related to the preparation of researchers in 

all three fields^ and there was a positive nnationship beween the 

preparation of teachers and other practitMners only in history. 

If ' 

This same pattern of relationships was f^und when student mean t 
scores on the program emphases were correlated with each other* 

Apparently historians Interpret the term "preparing college 
teachers^ much more specifically than do chemists or psychologists. 
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ud In a way that Is more like the chemist's view of pTaparlng 
SraduaMa tot ^ppUad earea» In industry or the psyeholpglit's " 
vlav of tr^nlng elinlclMS. The primary employers of the most 
outataadiag' seholars uong umv Ph*Djs in all three diselpllnes 
vlll ba tha najor imlvtrsitiea of the aoimtry TOd, therefore, it 
wouM not be inappropriate to pay some attention to the preparation 
thaaa parsons for teaehlng. However, very few doctoral progr«ns 
in ehemistry or psyahology set out to train teaehers for aontlnuing 
careers in imdergraduate or eonmunlty colleges. Some history 
dapartmants, however, do make a ipeclfic effort to train teaehera 
for careers as mdergraduate or even seeondary school teaehera , 
in addition to preparing scholars/ teaehera for mlverslty positions, 
•nils representa a different and less researdi-orlented kind of 
training, juat as the preparation of clinical paychologists Is 
differant from and generally not considered to be eonalstent with 
tha training of researchw^ra in psychology, 

Corralatas of Frogrm Purposes 

What tends to characterize the departtents which place a high 
or low degree of emphasly^ m each of these purposes?, The answer to 
this question Is Import^B because we need to underntrtd the 
antacedents and effects of the differences mong pro^rmfl with 
'dlf farent purposes mf because such information may be jseful uo 
programs in which changes In purpose ara being conslderea. Only 
gneral patterns of relationships will be repor^pd here, ruther 
thm a lot ^ eorrelations. ^e correlations are reported 
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separately for each of the three dlseiplines. Table 3.2 lists 
corralatlmis hatvean tha faculty lnda%a8 of progrm mphasas and 
aalaetiid progras daserlptors» The measures of these program 
§iiaraat«ifflstleB are described In detail in later chapters. 

fa€U^ty p^rL^aptimis of their departTCnts' emphases on preparing 
reaaare^arfl ware related positlYe^v^d strongly to the reaaareb 
activity md productivity of the faculty , the acadefldLo ability mid 
scholarly coimitmant of thti gtridentds the prograi^s emphasis on 
twining seholars rather than craftsmen p the eoploymant of aluni 
in Ph.D, universities p Bnd both peer- and self-ratings of the 
aeholarly excellenca of the program « T^ere is little or no 
relationship between a prQgrW'S emphasis on the ' preparation of 
, \^.ircheY8 and students' vitiws about the quality of toauhlng in thu 
V rogram or their views about the quality of its TOyir^nment for 
X'-Arning, 

A13b*^f these rorrelates 8U|ieit ft high degtea of eorrespondenee 
eWeen the emph^^^iB on preparing r searchers «id the activity mtf ^ 
atmosphere of reseat in the pTOfrains. That Is ^ faculty in 
de^ artm^'ts wnich mm^ asiie preparing researchers, are the^el'/es ^ 
resiiarche^s ; the faculty and students in the programs emphasising 
this putpLne were oriented toward scholarship p and thi^ alumni 
damoniiiat d n research "^e.^tatlon in their accomplishment 8. These 
programs U 53 t:ended t i he large ,^ we ll^paylngp md well^eqiilpped* 
n.^:,'i\v*>-r J they were not nt^cessarllV congenial or pfsrson^^Ily vary 
satlyfytng places for students to study. 
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Corriiatis of FicuUy PbEQiptioiiS of Progfai Effiphiisi 



■ Corralitioni with Cotff ktloM with 
iiphasis on training iiphMls on tfilning 
resiarehers in* ' tsiehirs in* 



JS 



C H F 



C H P 



1, Ficulty lilitfdi ictlvity (F-Sc 13) 6? ?6 6? 

2, taltf jouail articlss psr year 

(H-ilfflda). 80 48 3? 

3, Studat sdslarly cdnBitiint 

' (l-Se 14) n 57 70 

4, Mmt ef, slumni aiployed by 

Ph.D, wiivirsitiii (A-iy) 78 W 38 

5* Inphyis on triinini icholirs rathir 

thfe'i i'^aftiitn (F-l*o) 
L Meqvicy of phygical and finuicial 

viieurets (f-2'x) 
h mimm for learning (S-Sc II) -05 -42 10 
8, P^tulty concem for studints 

^S'Sc Ik) 
h :;^lity of Ciichini (S-Sc 17) 
^■u litudtftt-rilid scholarly exeillmce 

o! ptogtii (S^Sc 1*2) 
Caculty-ratid seholarly ixcellince 

of proiraffl (F-Sc ll) 
L2. Almmi-ratid sxciilsncs of prograi 

(A-ic 12) 

13, PiiMited quality of the faculty 
(F^'iS-i) 



73 32 64 
63 71 61 



■05 -32 28 

13 -02 61 

77 42 72 

83 86 76 

)k 85 69 

87 87 75 



23 -41 41 

29 -01 34 

34 05 55 

19 -10 22 

48 -12 14' 

36 -27 45 

31 49 1? 

20 58 36 
.05 38 46 

33 11 39 
60 -23 54 
51 -11 11 

34 -59 28 



T 



Corrilatioas with 
asphaiis on trainlAi 
airs in* 



C H t 




-14 -23 -60 

25 02 -21 

21 12 -02' 

ir -11 -52 

46 -27 -61 

44 -50 -54 

* * * 

-21 -55 -63 



Wilitions an within diicipllnes, basid m dipartiint nans. DisclpllBSS ats ibbrivlit^i 
C • Cheadltry (H»24) ; H » Hlstoty (N-25) ; P • Psychology (N«24) . 

h, thii tibli and Bost iuaquent onis in tM riport, ths seum of sieh Miiuri i»iif ^^^^^ 
thi vffiibU libii. Individuil itii nuibits froB at ficulty (F). sludint ( )t fj/ f 

^refill (DP) qusstlonniiris in iBdicitid; copiss of ths qusstioanairis ari in Appendix h Tht it^ 
10 fori a scalii icow (Sc) are llitsd in Apptadlx F, 
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Carfelatlgos be^een raphasls on training collage teachers and 
other progm diaraetsrlstles tend to be som^hat less high than 
for fMphuls on the preparatlcm of researehersp suggesting that 
there Is worm variation from progratn to program. However^ tbete 
it a pattern of positive association betveen raphasls on the 
preparatlcm of college teachers and reports of good relationships 
betvemi students and faculty In all three disciplines , ud a positive 
association between emphasis en preparing teachers ud the research 
and s^olarly orientation of the ch^d^stry snd psychology departnerits 
but a negative association with this orientation in history. These 
findings are consistent with the view es^ressed earlier that progms 
t^ train teachers should be staffed by good teachers who glve^ 
partleular attention to the quality of student-faculty Interactions, 
and that "training teachers" usually Is perceived by chealsts md 
psychologists to mean training researchers who snay also teadi* 
However t all b^^t the most prestigious history departments tend to 
train hlsto^lms specifically for careers In teaching. 

Very few .program descriptors correlate^ positively \.4th faculty 
perception of a progru's emphasis on training other practitioners 
in my of the three disciplines » suggesting that the characteristics 
of these programs are not very well defined by the variables in the 
study. Faculty meters agreed that these progrms placed less 
enphaais en training scholars than craftsmen » Md the students were 
more likely to have postponed graduate study and begun the progrma 
at a latet age. Students and faculty memberr In the prograraa as 

' y 

78 

t 



3.19 - • . 

irell ai assleaal psars parcalva doctoral projrws (l|^t place 
great on preparing graduates for applied poaltl'ona to be 

less s^ojjirly thn programs with other emphases. The negative 

relationship le perhaps strongest In psye^old^y^ hut Is eharact«r-- 
istlc of all three diselpllnea . 

This aa-^Htlonal evidence that prograois to train professionals 
for applied careers are not given high ratings on scholarship even 
by their am partlcipMts may be saying that the research eKcellenoe 
of tht faculty amA the scholarly commitment of students really are 
not the appropriate criteria for judging these prograas* These 
responses ©ay reflect the position that practitioneia^ not scholars » 
are needed to train practitioners i bright students inteiested in 
applied problems or service careers may be more appropriate than 
brilliant students in pr*3grams desipied to train practitioners* 
The lack of positive correlates uay simply reflect ^he absence of 
appropriate questions oryoptions in the data collection * 

Implications 

The different ways of looking at doctoral prdgm purposes as 
perceived by faculty members and advanced gradimte studTOts-^^the 
Ij^ortance placed on each purpose by the progrrap satisfaction of 
the respondent with this emphaJiSp and the compatibility and 
correlates of different progrMi purposes^-conslatently Indicated 
that the major purpose of the progrMs in this study was perceived 
to be the prepuatlon of sdiolars wd researchers, with more or 
less secondny attention to the preparation of college teachers- ar I 
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prefasslraals for othar applied practice * The preparation of 
reiM^^srs aiid taachers was p^arcelvad to be compatlbla In chemistry 
mA pBytixolQgyt apparently because many faculty m^bers In these 
fields Interpreted "teaching" to mean researchers ^o may also tea^. 
In history » however, aomm program? gave specific attention to the 

preparation of teachers (In contra&F; to scholars who may also teach). 

i 

Preparing teachers In history was per elved to be more like the 
preparation students for applied careers In the other two 
disciplines and» therefore » not positively associated with ir Jicators 
of scholarly excellenc^^ 

In SOTSiary^ It appears that the purpose of training researchers 
is the only goal given wide recognition by participants in doctoral 
progrms anC Is the only goal that is well-defined and well-imderstood 
within the disciplines as well as consistently related to many 
aspects of progr^ structure and ftmctlon* Increased ^^hasls on 
the training of practitioners » and on the training of teachers for 
undergraduate college positions » is perceived to be desirable but 
not very compatible with the reaaarch emphasis. The dllOTsa of 
quality vs. diversity is all too apparent (Brenemanp 197 3 i Clark, 
1976) . 

Unfortunately » though this rf.;iaarch attempted to be sensitive 
to differing program purposes, ir ^^^s uot successful i^ Identifying 
and measuring positive and generalizable characteristics of importmce 
to programs that emphnsiied the preparation of practitioners * Many 
of the characteristics that were measured and that are reported In tha 
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raidodw of this report mrm most appropriate as descriptors of 
programs that e^phaslEa the praparatlon of rasaarchsrs or . 
reaaar^ar^teaaUers . Bils is inavl table, given the high level of 
ai^hasia ^algu^d to Lhm p^uyAt^Liua ui' r^Mt^archar^ by the faculty 
m€^ars of thci pfograma In the study* It me^nes however ^ tnat 
cha f^ programs that did emphasize the preparation of graduatas 
for applied caraars had little chmce to indicate their strengths. 
Information In addition to indicators of scholarly esceiillanee is 
needed in ordain to raake sound JudgL.enta ^out these programs. Some 
poaalbla Indicators in other areaj are suggested in this report; 
others need to be developed « 
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Chapter 4 




chapter '^i^amines informailon regardirit 
usefulness of departmental ratings as indicatora of prograir quality* 
TWe kinds of ratings information pra considered. The fir^^t is pesr= 
ratlngS"tha ratings of one department or program by people in oth^ - 
doctoral progr^as within the same discipline* Then self^ratlngs are 
examined; that is, ratings of the d^partroent bv tlioee who are or 
have been in the department^-^nameiy , the faculty ^ students* and 
aluroi* Both 0/ thes^ ^Inds of ratings are very general ratings 
of the quality of d^^.partmente The p. er ratings in this study, 
ns ^^sll as Ir earlipr rMiHIpr. are ratings of the quality of the 
faeulty in a given depa nent. But unev are faculty ratings 
only in a very general sense 3 for they do not seek judgments 
about specific faculty qualities or behaviors, such as quality 
of teaching, Qriglnality of research^ adequacy of clinical skills. 
Similarly, the departmental self-gratings are ratings of the 
"scholarly excellence' of the pvogram, a quality that is obvlouely 
a collection of many distinct charact . lj^ lIcs . In effect, then, 
both sets of ratings examintd In this Jiapter might be termed 
global ratings of department quality. ; 

In view of the previously eKpressed restrvations sbout peer ratings ^ 
especially whin uped for individual program evaluation, one migh^ 
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wonder why pter ra:ings ...r ^ ccli.^cted at all. ^:'int it 

should be emphv^^i/ied th* n jur reserv, v Louh ah at pe^r ratings ai 
really reservations about their [ise as the criterion or eKclu^ive 
definer of doctoral program il^ty* As one of many indicatory 
of program quality j howe^'er r atings are clear "^y relevant and 

deserve attention. The/ wo i^deri in chi5 studv for thrca 

reasons^ First, we wanted o a i rter unders tanH . ng of peer 

ratings as one of man'/ inJic-tnrs of progran. quality^= _:ieir 
reliability, their rOiaL: . ^nsh . w;,th other progriTn descriptors, 
and so on^-^in t^;^ sai. e way hat ^e nought a better understandi' 
of the many o^^her progrAm characteristics explored in this rep^^t. 
Seconds we waiiLed Lu Laktr ayvanL:ig^ of Liie upparLuniL) tu cunduct 
a partial replication of the e //"E studies, paying particular 

attention to the stability of peer ratings over time and across 
different groups ot raters* Thirds given the widespread use of and 
familiarity with the ACE quality ratings ^ it seemed important to 
obtain a current similar nieasure which could be used as a sorting 
or cjassif Icatipn variable in analyzing the data regarding students, 
faculty members* ^Inmni. and denartnnpnt^ . 

The last of these three re^isons-^^'to use peer ratings as a 
classification variable in analyzing other indicators of quality*^ 
has been discussed in Chapter 2 and utilized In the previous 
chapter on program purposes. VirtuaiL.y every chapter eKamines the 
reli tlonshlp of. other program information to peer ratings* 



The first twc reasons citeu rfove^ — to gain a better understanding 
of peer ratings and to compare them in st»veral ways with similar data 





4. 3 

obcainad in earlier ACE scudies' expior&d in tr^^ chapter, 

along w.th self^ratings information collectec f rom /depar tinent ^ 
faculty, students, and alumni. The analyses of tiftesp data were 
guided by six major questions: f 

(1) To what extent e^re peer v .Ings stable over time? (That is, 

% 

to wha. extent do the peer ratings ia our study agree with 
tnj^e obtained in the - : • ^nt ACE study?) 

(2) Tc what extent do certain .nodologlcal steps^-e.g^ , 
eliminating ratings of one's o\v current department or tn^ 
department from whicn ch^ dn^^corare was earnad^--af f ect peer 

discifiline 

subspecialties within the - ame doctoral prograni? 
(4} What is the reliability of departmental self-ratlngs i\fornia- 
tion, as reported by faculty, students, and aluirmi? 

(5) What is the relationship betwe i peer ratings and self ^ratings? 
Do they essentially tell us the sara thing? Does this differ 
by discipline? 

(6) What is the relationship between ratings (both pee^^ and self) 
and other r!epar tmental characteristic-S? 

ina Nature of the Ratin ^'-s 

The peer ratings were obtaint 1 as fo^'lows* Each of the 25 uni- 
versities participating in the survey was listed alphabetically in 
the faculty questionnaire, an i the respondents we^e asked to rate 
the quality of the graduate faculty in etieir field at each listed 
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institution on a scaie%an^lng from i " t ingu itshed" to "not s\-^- 
ficient for doctoral training," Responcents wt. re also gi-.en .he 
opportunity to indicate that they did v.. it liava enough information 
to provide a rating* Thusy.-^the scnlo was idcnt=2al to that 
used in both the 1964 and 1969 ACl sur\^eys* Respondents were 
Instructed to consider only the scholarly competence and achieve- 
ments of the present faculty when making their ratings* Respondents 
were also asked to rate tlit attractiveness of each doct nrogram 
±± Lmdi \ UL Lite ^xacL woruxng ui Docn queSLions , see iten f^Z5 In the 
I acuity questionnaire ir; ApptriJlx DO 

Self-ratings were ob rained by asking three different groups 
of resDOndent s*"— f a rti 1 r V , Htiid^r^^^ -. pirsd alumni nf rhe H^nArrTTienf * n 
rate the department on a variety of character J sties r "atad tc che 
dimension of scholarly excellence. In some instanceSs respondents 
were asked to indicate their agreement or diBagraacnant with state- 
mants about the department (e*g*s "this graduate program is one of 
the bast in the field''), orj-in other cases, to rata cartain departmental 
charmctarlstics I subh as the scholarship of the fa ultyj on a scale 
ranging from sxcellent to pnor. Thusj the sellr^i atingf of the 
dapartmants are actually composites of ratings of fterent program 
characteristxcs s Jill of which are rel^u i to the jncept of 
scholarly axcallance* These separaLe items were hen comblnad to ^ 
form a ^ale score to represent the scKolarTy e'^ ' lienor of a program. 
The j.udpaentlssi-^^^uity^^^ students ^ and aluimt are each representad by 
a scala on this dlmansiohi Details ragarding thas: scales, including 
tha spaciflc items the intercortalatlon among the Items, the 



scala -e. ^ abilities , 3nd on ^ ar- pre-en^e^: ■ Apper. F. .he 
scholarly excellanc^ F^ales are StMieTit ^caio 2, FacuiLv :^ ale -I, 
and tfurnni Scale 

The Stab ility of Peer Ratings 

The 1969 ACE sLudy (Roose and Ander---nj 1 970) compared the 
reputational, i._ings data obtai-^c : i.. their svjrvey with those 
obtained five years earlier rrartter, 1966;. The t atings were 

that were Included i boLli studies, the correlation between quality 

ratings exceeded .9U in 26 crises and was below .80 in only onii 

instanee f 79 for Phamacology) . 

Wc ware able to comoare the mean ratings of the pro^ T-ams in 

1 

our stu^ with those obtained in th- 1969 ^tudy. lese data t-re 

prasentad in Table 4.1 and pff*rir clear evidence that, in ganaral, 
ratfngs of faculty quality are extremely arable /er tima. The 
new rating*- cr ii-t of ratings by all faculty questionnaire \ 
raspondants from tacn or universicieB xn the aLuuy,wuu wek^ 
askad to ^ only programs^^ in those 25 universitiee • The ACE 
d^^ta WEi.^ ..elected from a smaller number ot faculty members 
at a much larier n mber ot universitle^ who rated almost 

\ 

avery ?h*D* -granting progiram in their field. In this 
respect I the ratings obtained from tiie two stuiles are not . 

h^m wou? 1 like to take this oppurtunity to thank Charles 
Andarsan ci the American Council on Education for making the 
1969 ratings data available to us f(5r thi ^ analyBls* 

"\ 



1975 E7S Stud:^ and 1969 ACE -tudy 



Chemistry History Psyche logy 

Rankings : nk i r^s ; Rij,.ikln; ; 

1975 1969 1975 1969 1975 1969 

(ET-) (ACE) Diff. E' 



a 

Dert 



12 2- Q 2 

W 3 -1 W ^ 

Q 5 5 - r 5 

F 6 ' - W 6 

M 7 9 +2 D 7 



G 9 

D 10 

Y 11 
X 12 

V 13 
I 14 
B ij 
N 16 
A 17 
0 18 
K 19 
C 20 
L 21 
J 22 
H 23 
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+2 
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J 


22 


21 


-1 



RHO ^ ,99 RHO ^ .99 ^ RHO = .98 



^l^ntif Ication letters wee asFigned randomly to each university in th^i 
study. No unlverslr^ or program Is dennified byname In this report, 

f 

Notei Somr: departments Included in t> ^ 1975 study were not rat^^ n the * 
19W AfjE survey. For these departments a comparison was therefore 
not possible. As a result, the number of departments compared are 
as follwa.: Chemistry ^ 23, History ^ 21, and Psychology « 22* 



by the high rank order ( jj cor re 1 it i o:, ^ , In rVo^i--ry, wb^r^ the 
correlation between the 1969 ACE ratings dnJ the current ET^i ratings 
waa *9^| one department's racing v/a^ chre^ r-anf,s aighiar in the 
current study tiian in l^^o9 (DepaiLmenr Y) , anoihei^s mean rating 
.-an three i unks ' ower ( f' Lnant ^nd ail others VHte eithc^r 

' iln one or two ranks. Tn hj^scory, where the cor- 

^.f. . ^- ' ' - ju bv Hior^' tha'^ tu dIuc^s :a feicher dlT'. wticn. 

.-Iv .;jrti cnti cc.:tilatic)4 uetw:^-^ rati ig^ waa .yB, 

r?3j.arcmen - ^ ^ac a i:.. proved th^i ranV ings by four places, thf. 
r.nnkingH r ' - v-^ tr-^ntb D an^ decrBasert by three places ^ and ajl 
ethnr r&nKir:J p v.f.r%^ e * ner identiCf^I or within one or two rai>Ks . 
ThuSs evan t/iough the overall agreem* nt among program rankings over 
the six-vear span ii dramatic, it ca" be snen that there were several 
sizable Indlvidnn' pragma shift AUbnugh reputational ratings 
f.ra dfcf initaly >atabie enough over turterent raters ana over time 
tu peruiit us to be confiient tha*^ the^ ^ i reliable, at the same 
tim the data cle. rly sdpj-est ' J e ^re some changes in the 
ratings ^ >ilvidual orogr-ams. 

• 

Prugram Q i Ly Vc4. Program Attrac tivaness 

Bctr. or the AC), surveys asked faculty re&puitdant^ i.u rat^ two 
saparatk^ .,spt^'^^^ of graduate 'jr jgrams* the quality of the faculty 



\»^erB rated in Ijth ^Luait^b ib rh.n^tju (.uily - I' '\l ^ , iiiuic^iL^u 




^ zie attractis^eness the doct^: . a^. r^io^.-.-n/ int^^ ' irM.e;:5 . 

7or each of he 36 disciplines includ. im. ''6^1 cuuy, ^he 

"^^Qrrelatioas between raculty quality ratings ;:\v\ progrn'i atc^actdve-- 

nes^ ratings are either ,98 * 99 . Thrs relation >ij: betT^een the 

tvd( questions was also hl^h for data obta:^.ntd i c; the ^^64 survey, 
s 

tho^igli no correlat ions w e - e rfe-^orte. . 

I i t^Qtt^ in Table 4*2, Lhy reia:ion j b.i woen the faculty 
quail tv rntin:. : pl^A rhr - - - r " * r,: ■ " - .-r^- - rn-)n,:n n>ifn1ned i 
-urrent JsPidy ^as alsc very .Jtrang. ThiS ^cr- ^iations * reported at 
':he bottom u/ t^^' L^'^lCj arc^ ^^9 ul^ th' \ lisc i=plinea . In 

chamlstry, oniy one departmeDf > rarki:.g w^s different by more than 
one plr^ce -^n th^ two ratings (Depar.; s t X, whose attractiveness 
ratin^, *^as lw^ hjiks highfei). 'n histury, lu^ ranks of three 
departments diff ^^ed by thr. 4i places (D^partiaents C, and L) , and 
In t^yychologyi DepartTnent T rrc'?1ved an ^ tractlveness rating that 
-^^n four rank- lov;- ^ then Lhe rating of the quality of Its faculty, 
in cne grea\: majority oi c^ leb ^k. aii three fieidSj howevtir , lii^ 



2 

Actually, the instructions to respondatits In both AUi*. surveys 
In'Ueatei that they were to rate the ef f ectlvfeness of the doctoral 
progrn , In fact ef fectiveness Is the word both reports use when 

rftfarrlrg to the data. ^owev jr the respense options on both qups-- 
xonnaires we Actually differijnt points on an attractiveness scale 
raiaging from extremaly atcrac^ve" to "not attractive")* For this 
raason« a: d "he feeling that raters would often have no basis for 
Judging a p. jgrcur'a effect:' veness bat could offer their own toprea^ 
olons c ; It attractiveness ^ we opted for "tractlvmess as the more 
appropriate and meaningful of the two cont^^pts, for the specific 
warding of Ins .ructions to respondents* see item #25 In the Facvltv 
Queatlonna:* re In Appendix D. 
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rankings ^ased o.. i ^-ir* ivRn^ T^^ir^^r -QrQ -■ f - icrrnticai. v'i^=-^ 
within nne rnnW of * he rankin;^ ^ " "fli;: ' of uhe au^litv 

Che faculty. 

It would ther^^ore app tna r»e racern wr-re either unable to 
dj^tinguis^i between the qua] i a program'ii ' icuity and it& 

3t tiactiveness for graduate students, orp mor^ l^cel-^. that the ' were 
able to aefe ^he ^^oncepLuai dldtiuctiuL but simnly felc that the 
most Import ar; L ^ ^ fa.jl r : n Uetermlnir.g a progrn'^^ >5 attract ^ ve-- 

nes^ for scudpnt'-; Is the qu. 1 it ■ of the raduate fac^ Ity, In eii ler 
case, the data su^gust that onlv one cl the l...o measurtzH Is n^^ :SSAry. 
For mos t r f the a: i a 1 v s es in i. h i s r en n r t r ha t have to do with peer 
ratings, ha% j chosen to u^^c ratings it th^ quality of the faculty. 

The Effects of Inc luding Rarlngs of One^s C>ti Department 

In making thf^-' i r^u ngs of the quality of the faealty in prograins 
in this study* -^e-aondents were instrucued to rate the faculty in 
'hair field at ^ach of the 25 institurions listed. ^These 25 depart^ 
ments obviously included the department n which raters ware 
currantly employed and, also, the department from which soma ratars 
had earned a doctorate. An important question » than, is tha effect 
of these ratings on the overall ratings of the programs. Data 
relavant to this question are pt ^dnted in Ta. le 4.3, In this table 
the mean ratings for each of the 25 departments in r>^cr of tn.; three 
disciplines are given for three differer rating condltlonb: 
(1) those which exclude atlngs of both one's current ampl^ ^^g 
department and the department from which thti ioctorate was earned 
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^Firbt column of ra^inHs wiLhin t ach discipline are clean" ratings; that 
is, ratings which sxclud? Loth curr-nt or^pi^jMng ctepartmpnr and nepartmpnt from 
which doctorate wa ^ earnr ' . 

^Second column of ratings within each disciplinp inclndu departmant from 
which dnr'urate was earned, but exclude current enT" jying departraent, 

^Tlilrd CO UTOi of ratings withia eacl iscipline include both derart,..e>rt.^ — ^ 
from which doctorate was oarned a i curi Tit employ. ^: ■ partroent . 

*These departments are oncR for which there a difference of .10 or greater 
bl%fp/een cols, 1 and 2, 1 and 3, or 2 and 3, See ^-t for elaboration. 
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which docroi its was earned .n c^I mr twOy , and (3) ratings v^. " a 
include both current employi: .epartment and the departmant froni 
which the do'^toratG was earnud. The ratings are baHe' on a 
si.: point scale ranging from zero {not sntricient to award a 
doctc ite) to tivt' (distinguished) . 

n 

.L is appai air from labia 4 3 the. wh^tb^i ona'^ lurre^.t : 
-Ti^-nT n Hr-p^rrmnf f^^nr ^^hlch th^. doctorate was earned are 

rem^.ed .>Qm t Included in cue analysjp makes little difference on 
^ther ab^ ^uLe r.itlngs ' tl- means) or the relative rankings of 

programs. Across ail 25 departm' atSi the means the ratings * ! chem- 
istry vary by onl> ,03, from 2, when both currant and d^gree-grantinL 
d^ya^t^nts .ire ..:cludcd j 2 . 39 vher nait-er Is e^^-luded. The 
range in maans acro is these l'. ree ratings conditions ^^Tnewhat 
grea ^r In history (from 2,78 to 2.83) and psychriogy (2.95 to 3,u0)_i 
but in no of he thre.^ disciplines is the dif^^irence very large, 
rtiuu iespe^u iiidivA^tiaA prw^^a^iTi^t TiCfe crsC ch£S^>^tr/ ^r^grs 
varle by mnre tha^ ,0^^ across the three rating conditions. In 
nistory , five prog meanb van d by * 10 or mute (Dep«tLm^. 1,b 
1/ V, Jj and H) , and in psychnlogy three department maans varied by 
.this amount (L^partr its V, H, and u), 

'\te that the fi^^ histr-y dtepp-tments wlfh any appreciabla \arla- 
nlon in means acrosr che ratings dondltlons and the three psychdlogy 
departments with any apprectable variation are all icpartments 



wuose ratiugs ten-: * wi: i tn^ l-r-v ■ -.i of --^---^^ /i^ s t r i b'-i: ion 

p-- "=ratings tend tn rate hei- jwn depar* Tie:^t faculty sli .tly 
hi|?her than do their feers ' , otrier departments, whereas the se^'f^ 
racings of fact icy In hie " y rated r "-ograms are v- rtually identical 
with those of theii f-^u.s in other programs. ^ \. Is also Ir ' eresting 
t ) note that tiie mean . f t r.^iCings oT p vch jIog.TSts Is t e highest 
among trie chree disciplint»j rollowed by the ratings of cher Lst^ 
and historians, in that or*-.er 

We have already Indicated that ratings undei each the three 
c ndltl^ns nre very similar "^tillj of the two p sslble sources of 
bias in che ratings — iucludiag t\ department from which the deg.et^ 
was eamc and including the raters' currc t jAploying department - 
it would 3ppe t^hjt \' letter source of bias Is of more concern* 
The ratings that exclu.. Doth c^^rrent and ^egrep-granti.;g depar*-:nents 
(in Column 1 in Table 4,3) are vi: cually identical with the ratinge 
that exclude only the currant employ ig department (In Column 2). 
It ia only when the r..;in£^ in::luic the r-Lcrs' current . rlcyin- 
department (Colrmri 3) that differerces are observed, ( nsequenrlyp 

ChrOUgtiUUL Uii^ tepUlL UUi r^ual^^^s ^i. pt;e leiLlu^s will Itifer Lw 

ratings that exclude the current employing departnif .t, but Include 
ratings of the deparLment from whiun the docrorate was earned* 
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One of t critlci--^^ ' D^^er rat:.ng" i& thau manv those 
making the ratings are ^urtir'^ntly - with many 

if the programs to be able zo n u.a more than a ster^fOtyped gi ass 
abwut the quality of Lhs faculc% The rL'G'iest for ratirgs of the 
quality of the faculty at the various departments in the study 
gave each respond en t the option ol" ^ndi eating that he or ;=.ie did 
not 3ve cnoug.. infcrmacion aboiir a department to offer a r^, ^ng. 
Me^n quaiity rri Ings were then based onlv o.. t1" i responses of those 
who mau^ a rating and wh(? therefor- ; resuma. ly fel'- tha, they lad 
enough information ^hfut \ e de.^artment to make a jpigment. 

Information abcMjt h ^lationshlp bett.-^en peer qual ity ratings 
and the number r^-r is p res en tea Table 4.4, whei ■ departmenlB 
are listed In the order o the rated qualitv of their fh ulty. 
As KJiie wuuld expects tne t imbar of raters Is largast for the depart- 
ments with the highest rat:^ngs and smallest tor the deparrments 
with the lowest ratings all thrt^ ilelds. The n^bei of raterf* 
r^n&^t^ frnw ^19 m >4i in rbeTniscry ffor Der ir^ments U and R) * 
482 196 to history (for Dnpartments P and H) , and 5C1 tu 186 in 
pcychola^y ifzT Dcpartner:"!^ U ^md R) . In all three fields the t^.-^ 
10 or 12 departments were rated by 400 or more respondents * but 
thereaftL. t\\Q numbe: of intormed raters drops off rapidly* 

Thase data make it clear that a largt number of raters refused 
rate dapartments when they felt th?it th^' did not have sufficient 
information to do oo. The much ler number of raters for the 
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Psyrhology 
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At the saiiie tinuz , the ?:otal n-:^b^-r rartni^^^ -r^i thv-se duD^rn-^eni:;^ 
is sufficiently large to pr/r^id^ h rrjlia^lo -riean ra'in^. No- th^t 
Che standard deviation of the rat lngS'--^that is, thti . tendency for the 
ratings to vary —was not consi-: cently gr*satef ror the dapartmants . 
with the smaliast number of rarers than it was tot departments with 
the large&t number of raters. N'aturally, w ^en ratings are batted on 
,very small, numbers , concnrn abDut these matters is Justified, In 
this case, however, there appear ro have been enough raters of 
even the least prfiKtlgioufi departments to warrant a confident judg- 
ment that the ratings are very stable. 

P rogram Subs p ecialty R atings 

In spite of the many atudie , that have been conducted on the 
reputations of academic department c in graduate education, we were 
familiar with no previous attempts to examine the raputational 

ratings of departmenta? .subspecialties . Tha^frequently .referenced 

' ^ ■ ^ y 

ACE studies asked respondencs to rate .departmental quality * giving 

no attention to the possible variatio^ of quality among the various 

3 

soibspecialties within academic departments. Since all faculty 
s - 

members in. the subject pr.>grams were invited to make ratlrgs in 
this projects providing a large and heterogeneous group of raters, 
we decided to obtain faculty racings of siAspecialties within 

^ 

' 3 ' ^ 

The 1964 ACE study (nartters 1966) did examine total progL'am 

ratings In English depart .ents separately by raters whose area^-i#f 

specialization we* American Literature and Britlteh Literature ^ but 

the iubspeclalties themselves were not rated. 



98 



departments wtli a& oi torai ajpiv . ■.Hi: ■.•Mrp:;^wr w L 

department subspeclaltieB varied and , a.: d" ng / . r - sxtt^nr to 
j^hlch the more ^^eneral peer ratin^ri are sutninari^-s baded en 
he.te ^eneous or hoinogenecn''^ wl^iiln-departmerit reput tions. 

The faculty questionnaire ; ked respondEnt: > to Indicata tht ir 
dlBclpline and the one subspecialtv wl ch which they identliifed 
r.ij^t closely. The ni^rner of aubspecia ; ties varied disciplina 

Then respcndents were asked to rate bo::h the quality of the facuny 
a;ui the at:tractlvene.:4o oi the rr^^Ki^^t'^ th^^ir own subspecialty 
at the 25 uni ver i^iLies in rhe study. The forTnat of the rc:quasted 
ratings was the eiame as :hat used tor the verall program ratings , 
with the laculLv uf sub^^ueelalLle^ rated as being distingulshad * 
strongp good, adequace^ margi nalj or not sufftcient to cflimr doctoral 
training. Respondents were also given the opportunity to indicate 
tbdt they did not have enough Inf om^ation about a subspecialty to 
mafee a racing* LnBuf^icient iniormaLiuu pretjum^bly ^lau Lhe 

appropriate response if the subspecialty was not offered by a 
particular prograiti. 

The subspecialty ratings data are suranarized In Table * 5, ^ The 
firFL thin^ that is apparent-^f rom the ratings of faculty quality ^n 
subspeclalttes Is that, like the ratings of all faculty in a progrM, 
they are very hlghA / corralated with ratings of program attractive^ 
ness* As reported in Jolurn 4, the lowest correlation between. 



■J o r r e i a 1 ' " n s of s ub^ ripe^'ialLgiiti ori 
faculty ratings with: 

(1) (2) (3)* (4) 

Range in Ratings of NiLinber Subspecla3 Izat Ion 
numher all far Itv attractiveness 
Subspaclalf leB of raters^ in dep*. , raters rstlngs 



Che mistry : 

(2) Ansiytir HJL lj-jj '^w . / j -^.v 

(3) Bidchemical 11 J9 ' 94 22 99 

(4) Inorgaaic 35-63 94 ^8 99 

(5) Organic 80^121 98 "^9 99 
ie) Physical 70-151 98 55 99 

(2) Ancienc o-23 ^4 32 ■ 97 

(3) M€Qieval 15-35 91 ?1 ^ 98 

(4) ModePfi^ 52-131 99 . 48 99 

(5) /-larican 75=169 98 15 99 

(6) Third-World 24=68 95 80 98 

Fvyehcilogy : 

(2) Educational 1-4 79 18 88 

(3) Measurement 1^6 91 22 9? 
, (4) FeFS^^iiality 6-19 88 66 97 

/5) DevetuDmental 9-40 85 3^ ?9 

(S) Exps^^imental 64=168 98 45 99 

(7) Organizational 7-18 5^ 77^ 97 

(8) giitilcal 26-97 '78 50 9:, 
fQ'^*So^±9A ' ^1 = 77 R8 61 '99 



Kefers to greatest and smallest number of raters ifor the 
department 'or subspecialty. Thus s for examples the biochemistry 
subspecialty at one department was rated by only 11 raters, whereas 
in another department that same subspecialty was rated by 39 raters. 



'■for Educational PsJ^chology) , 

It was also truv of all sub^p- i alt Les that quai:i:y racings wer^ 
highrr wheri there were larger ni^bers of raters (note cousistcntly 
positive correlations in Colunn 3). Inis simply nH.a..H that prugr^ms 
which received lower subsRecialty ';atings tended be t^ ^3 Insser- 
knoim departments that were rated f ewi- r V^^^f ' ^ 5 whareas dtpartments 
wi^h subspeci;?^ r --^d by mo^^t respondents tended to be better^ 

known and more prestigj-ous ht^^extent to which ti;U ^as true variea 
among subspecialt ir s , but wa^^ nev^r theil* tni*^ ' or a] L= 

With respect to the cn*^re 1 at ions between overall ratings of 
progr^am faculties and ratiigs of Lmbspgcialty faculriep, these are 
reported in Coluriin in histf ^^y^ all subspecialty faculties were 

rated in very much the same way as the total proi?rani faculties were 
rated, as evidenced by the fact that all intarca'^relations exceed ,90, 

the ayes of historians, then, subspecialty faculties ^ that receive 
high quality ratings are virtually ai^ays found in dapartments with 
^trwg ^'^y^i^s n fartil/lRM. In rhpK ^etry rlppartments 'a hlfh relatlDn^hlp 
batwaen subspacialty faculry ratings is also the u ual case, but there 
is ont gl^^ing £xccptiGn--analytical chemistry, Al-;hpugh there is 
a tendency for the more highly reg^ded analytlral chemistry progrmns 
to be found In the 'higher-ranked chemi&try department 3, evidenced 
by the positive ,40 correlation between the total faculty rating 
and the rating of analytical faculties, obviously * ..here are some 
notable excepticns to thiB general pattern in tuls field, ^There 
are tls^ exceptloi^s for Organizational Psychology. 



Taken together the data in rabie ^^5 do not make a very good 
/ case tor attempting to improve tho precision of total prograEi 
ratings by including ^subspe^ialcy ratings. As already discussed, 
the correlations between subspecialty ratings and total department 
ratings are gGncrally very hish—i' wost case^ so high as to 
make the separate ratitvgs unnecessary* Beyond this, hw6vers 
there are ^averal logistical difficulties irherent In subspaclalty 
ratings that make the procedure even tnore questionable. First , 
th^. ^e is the probleir^ of the number o^ racers. As shQfwn by the 
data in i.ie first column or iable 4,3s certain suDspeciaitif s 

- ^ved ratiags by » :.iv h very limited number of respondents. 
In psychology s for example, the Educational and Measurcmant 
E?f^f^tj-ppr =^ q 1 1 i ^nniR def*Rr f TT^ef^ w^t^ ^Iven nnlv one rating, 

Oth^r aubspecialties w^re also give , ratings by a very small 
namb^r of respondents, sometimes beC'^use there simpl ' were not 

faculty members who identified themselves with the subspecialty * 
sometimes because those ^'n the same subspecialty did not feel that 
they knew enough abo. t pubspeciaity faculties at some of the other 
universities*^ In additioft, there is the problar rf the effect of 
departm£;ital ''halo'* on the subspecialty ratings, l^r^*-ably s^fte 
subspeciQlty ratings .were dimply ^tenslon^ of totfel depart^nt _ 
ratings, -That this occurred , is sufigested Fy the* /act that some 
departments received ratings for subjpecl&l^iiii that *..erq not 
available in their departtWnfR. 
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ljb_ c__R<sl lit 4.onshlp Betwern Self -Ratings and P- -^r Patlng^s 

As iridicated earlier in this chapte^, i ilf ratings information 
was also ob^;ained f^om faculcv, stu^J-ntS| anu alumni of ^.^uh «iepirt- 
mant. Various questlonna irf icems having to do with perceptin^'^''o f 
the program's scholarly excellence were combined to form thr^e scales , 
on€», for ■each groHp of respondent. > "terns included ratings of jaculry 
and student scholarship and research excellence, student origi allt^ and 
commitment j and ^he degree to w-sich the department was a stimulating 
place in which to study. (Again, for information about the specific 
Items In these three scales, see Student Scale #2, Faculty Scale #1, 
and Alumni Scale ^^2j all in Appendix F,) 

Table 4,6 presents tht intercurrelatijns between thesa three self=^ 



Note, flrpt of all, that each of th^ three self^ratln^ scales is falrj.y 

Yilghly porrelated with the other two self-rating scales. In chemls^.y, 

for exmple, student ratings of st^holarly excellence ( ^rograrns 

^orralattf§ *84 with program s^^holarly excellence as pereeived by 
I 

the faeultyj and . 78 with scholarly excellence as perceived by the 
alumlf (Interc5rralations among the .three self ^rating scales are 
enclosed in the broken-line triangle in Table 4*6.) In general » these 
cuf relations Indicate a f airly high degree of agreement betWMn the 
§elf-^ratlng8 of faculty, students, and almnn'f Th^ lowest agreement is 
fou.id In fclstbry, where . percept lonf of the faculty and aluBMil agree 
more with, eacL ofiier laan citTier one does with the perceptlans of 
students* 



T ^ * ^ TT & 1 *- i - -i -^j 



Progr.ir: |ua 1 it^ 



Indicator 



1. Srudent^rated scholari 
fexcellance of program 



excaliance of pro-am 



3, Mumni-rated ^rholarly 
excellence of rogr 



4, P^f'r i-itlngs of quality 
of facuHv 



Nv.Le; C - r:.emi3try programs) 

H - 1 iatory (N ^ 25 pfojrams) ^ 
P ^ rsychologv (N - 24 programs) 

'^4*Excillent , J^Good, l^Fair/ 

^j^ulstinguishsd , 4-Strong, 3^Good, 2^Adequat€ 





Correlations 


Mean 


S .D. 


c 






.25 


H 




2.89 


.25 


P 




2.93 ■ 


.35 






2. ft 5^" 


.42 


H 


54 — 


2.86 


.27 


P 


7';. 


2.88 


.34 


C 


T' 84 


3.14'' 


.39 


H 


&1 72 


3.26 


.40 


p 


73 75 


3.33 


.30 


c 


67 . 71 78 


2.86° . 


1.03 




47 82 74 


2.79 


1.14 




76 76 39 


2.9 ' 


.98 



IV4 



Besides t:ie fact that If --ratingH of L:;obi^ chree ^ups tend 
to be in general agreemantj the other important fact in Table 4.6 is 
that all th ee Belf-ratings t^id to be fairl\ - Lghly related to peer 
ratingfe. That is, :h' opinion-^ of the i aculr , dents, and alumni 
within a given department in general correspc; n to the opinln 
that same department as held bv- faculty in the sa^-^ discipline at 
other institutions. Not surprisingly, the degree of this agreement 
is lowest for student ratings , where the correlations with peer 
^ cings are .67^ .^7j and .76 in clieirii ^^y^ histor%^, and psychology 
respectively. As a ruiej tnera is less agreement abuut program 
ratitig.=:- — botk seif^ ^a.iJ pfcf^r=^ Laiilrigs- = in history deparLments. 
The ratings tmong hiFtorians tend to agree, but not so closely 

i 

^Vt^ ^^^=1^^^ f^f ^y^f^ ^^Y^^^ ^yj^ ^^^^^^^^^^^^ 

The Relationship Between Ratings and Other Program Characteristics 

How do progra^ ratlngs^-both peer^ratings and self-»ratings=- 
correlate with other program characteristics? The answer to this 
question is extrem&ly important, for it has enormous implicauions 
f or ^tic cvaluaticn dsctcral pregran ^quality* If, for example, 
program ratings— paet^ratings or self-ratings—ar^ conaistently ^wd 
very strongly asi^uuiatttd WiUi ca^ray tetibLlc^ that art gerierally 
acQepted as reisonahre indicators of program quality, then thara 
would be a convincing case for heavy rilimice on ratings information 
in makliig suOTsatlve judgments about the overall* quality of a 
doctoral program* 



chararceristics cnar nave o- n been cit as reasonable LndLcators 
of quality and hav^ received prtviuus at c en c ion m tne reSearcn 
literatura. In addition; however, wanted ti yae haw ratings 
information wac. related r.o several "nontradi tionai" iriicators of 
prograip quality^^characteristics of doctoral programs thar. a ;fational 
panel of graduate deans believed to be Important quality indicators, 
but which t to our knowledges have not previously been exa^.ined in 
any systematic way. ADrrelations of ratings with a few traditional 

summarized in Table 4,7, Nnte that the index of self-rating '^hosen 

for this analysis the facuity self-rating of program scholarly 
excellence * / 

Several interesting findings^ emerge Frn^ the correlations 



reported in Table 4.7. First, it Is clear that peer rat Ing^^ (given 
in the first of the two columns under each discipline) are rather 
highly correlated with several traditional indicators of doctoral 
program quality* In this respect p the findings confi™ those 
reported in numerous other studies (e,g,| Beyer L Snippe^ , 1974; 
llton h Rose, 1972; H^strom, 1971; National Science Board, 1969)* 
Furthermore, the peer ratings are more highly corriilated with the 
traditional indicators than are the faculty self-ratings (in *^he 
second coluinn) * Thus , to the extent that these traditional 
characteristics define program quality, it would appear that peer 
ratings con titute-a fairly valid measure of overall program 
quality, and a somewh^it raore valid measure t' an faculty self -ratings , 



Correlati.ons c'' i'eer Rat ings and -'a^rMil ty Self-Ratings with 
V^Tlous Uther Indicators of PrDgrarn Oualitv 



ChemiLC^y History .Psychology 

Peer Self- Peer Self- Paer Self- 
Ratinr Rating Rating Rating 'Rating Rating 



Traditloi al Cha racterist ics: 
Size (FTE ^tl^dents) 84 63 68 55 60 45' 

Faculty publications 

(last years) 90 68 50 7 kl 43 

Faculty research acrlvity 

(F-scale 3) 73 6: 78 62 86 69 

Ph.D. from "leading" insti- 
■ tutions (F-15)a 68 60 74 59 - 80 . 53 

Selectivity^ 24 30 70 71 -21 -29 



Nen t r ad 1 tlop ^ 
Charactsrist^ ^8 : 

EnvlrotuD for learning 

'I 

Quality of teaching 

(S^scalB #7) 24 00 13 52 72 

Exaellant ear ear preparation 

(Alimil* A-4-abc) 58 66 -05 05 39 47 



^"Leading" Inatitution reters to programs with pear ratinjs of 
3*0 (^"good"). 

^Ratlo of nimber of stu nts applying to number admitted* 



/U7 
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Hwever, though the correlations with these traditloTial indicators 
tand'te fairly high, it Is Important to note that peer ratings and ) 
self-ratings are* not correlated equally highly with all of the tradl^ 
tional Indicators of progr^ quality across all thref 41sei^ilpes. 
Peer ratings are very highly correlated with publications in* chemistry, 
for exMple, but, the relationship is riot particularly high 1ft hMtory 
or psychology* Ahd student selectivity is fairly highly wrrelated 
with peer ratings in Klstotyr^but not highly correlated in chemistry 
and psychology— where i In fmcti there are slight negative correlations. 

When we examine a few characteristics" of doctoral programi that 
usually are not measured— eharacteria ties * It la worth repeating^ that 
graduate deans indicated ate important ones to^ consider in making 
judgments about doctoral grogr™ quality— It Is clear that quality 
laLiu^s simply do not correlate In smy consistent ^Ay. Self-ratings 
^re fairly highly related to quality of teaching In psychology, but 
not at all in history; neither form of rating is highly related to 
the department's envlronmttnt for leamlug (wMA there are even some 
negative corrc ations)* Whereas peer ratings were consistently more 
highly correlated with ftraditlonal research-oriented characteristics 
^hmn were self-ratings the opposite seems to be the case for the t 
nontradltional characteristics. However, neither fom of rating is ^ 
very hi|hl|r correlated with the nontraditlonal" Indicators, 

: r% t 

Susmary of! Findings Regarding Proeram Ratings 

The focus of this chapter was on global ratings /of graduate 
program quality, including ratings madfe by those in mm same 
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disciplines at other univeraltles (peer ratings) as well as ratings 
by fseuAtyi studatttSi and alimni within fhe depattment (self-*r#tlnga) 
thm purposes of the ehaptet^ were to exmlne several methodologleal ^ 
queatlona with respect to peer ratings (eig,, their sta>^illty^ 
whether to exclude the rater, am departraents when rtleulatlng 
f^tlrigs, etc.)i to examine the *rfelatlonahlp betoeefi peer ratings 
and self-ratlrgSs to. considpr evidence related to the "validity" 
ratings information * and to of fey Judgnents about the appropriateness 
of ratings Information as an indicator of doctoral program quality. 
The basilc firidinga of this chapter are these i 
. — teet" ratings arp quite stable over time. The relative \^ 
ranWlngs of the 25 departments %n this study wfre vety 
similar to the ranklnge of these same departments based 
on data 'gathered six years earlier. 
"Nevertheless J peer ratingj do appear to be sensitive t^ 
some changes in individual dapartmenta* ^ough the overall 
tankings vw^re quite similar, ahifta of thfree or four ranks 
dld%ccur for some departments. 
— Peer ratings jof the quality of a department's faculty are so 
highly correlated with peer ratinga of the attractiveness 
of a doctoral ptogram that very little Information seems to 
be added by Including a rating of proferam attractiveness » 
^-In obtaining peer ratings, it seems to make little difference 
, whether the ratera* current emptying departments or the 
departments from which the raters earned their doctorate are 
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^eluded from the ratings* Neither the abBQlute ratings 
nor ths relative rmklngs af pragrami appear to be sffectiad 
to any extent by this logistical question. 
'Eating program aybspeclaltlea dpms not appear to be a very 
promising alternative* Most 3ubspee|,alty ratings "arc very 
highly correlated with total progrm ^ tings md thus 
provide very Untie BddltlonaJ informatldn* -'Furthernore , 
there are few potential raters Iji some i^ubspeclaltleip and^ 
the laek of famlltarlty with a nmber of' programs results 
In sq. few ratings that one must be concerned about their 
reliability, ^ \ 

-Self-ratings of the scholarly escellrace o^ a program by 
faculty, sttidentSj and alumni are In gmaral agreement, 
though the extent of a,jreement is not as high In history as 
it is in the other two disciplines . All three <klnds of 
self -ratings correspond fairly closely with peer ratln|S» ^ 
External Judges and' those, within departments se«i to be In. 
general agreement^ about th$ academic excellence of the 
faculty and related program ela|entSt 
-Both kinds of quality Mtlngs are highly related to progrn 
emphasis on preparing scholars /researchers, program 8lie,4 
and the training mi research InvolvMeat of faculty* Peer 
ratingsiare somewhat more highly correlated with traditional, 
research-oriented Indicators of program quality thaii^are 
seM-ratings, which appear to be somewhat more 'highly 
related to less traditional progrm' characteristics. 



4,29 



-Indicators of a ni^sr of other progtOTi characterlstlce 
provide Infon^tion aboit the quality a progrm that 
is not conveyed by peer ratings* 



Chapter 5 

^ FACULTY TRAININ.^ MD PERFOroiANCE 

Blscv^sloft thf» pr^v^aua rhapter concerning peer ratings nt 
the quality of grWusta faculties Is only one of many Indications 
that the ^araeteristies of a progrw*s faculty are among the most 
laportazit Indicators of a progrm-s quality « Fartleularly at the 
dotitoral level , it is often literally true that the faculty mrabers 
are the program: students enroll to study with leading scholars in 
their special areas of Interest , the faculty act as mentors as well 
granters of degrees, and they produce new Imowledge that advances 
the discipline while it also increases profassional awareness of the 
department. Peer rating^ of faculty quality are one way to measure 
this dlmensian^ In addi*tJon, this study assessed several more ^ 
specific characteristics of department faculties by averaging self- 
i^eports from members of the .faculty and ratings from their advanced 
graduate students. , , 

The assessment of facility character sties concentrated on 
three general areas i (1) the training, knowledge, and skill ^^hat 
facult^ members brought t6 the task of doctoral education in , a 
particular progrua; (2) the quality of the teaching by a progrjun's 
faculty I and (3) the sch^arly activity, research productivity, and 
professional contributlori of a department ''s "faculty members - 
Heasurfs of indicators in each of these areas will be tevievred In 
the first part of thds chapter, and 4 report of the ways in which 



they relate to one another and to a vrftiety progrra characteristics 
will follow. . 

|rj^lag mA Ability 

A brief proflls of.tha backgr.^unds fmculty makers who 
cotspl^ted quMtlonnilres for this stirfy will be "found In Appendix E. 
Almost all of the faculty members In every program (96% to 99%) 
mpc^rted an earned Ph.D, degree. Mi avera^^ of ^0% to 80% of the 
f^eulty members In each -program In all three dlsrLpJlnes had been 
rewarded by , protfi^tion to associate or full professor and held a 
tenured flppointment^ *A majority^ had more than Ip years qf teaching 
ei^erlenee. All of these charaeteristlcs of faculty mea^ers are 
possible indicators of their scholarly training and skilly md one 
would want to Inquire about them when evaluating a doctoral ^pgraa, 
but none of them offered sufficient varl^illty tp be very useful 
as a program descriptor in this st 



' Three possible measures pf the degree origlna <)f m^bers of 
the faculty were Investigated. Faculty questionnaire respondents 
* Indicated the university from which they received their highest 
degree t md the 19S9 ACE ratings of the quality of the graduate 
faculty In the appropriate field were asiigned to each respottdefiit« 
Three scores were then computed for each programi (1) the average 
peer rating of* prog^ws from which faculty members received their 
hlgheafc degrees; (2) the percent of faculty who received doctorates 
froB -'leading Institutions" in their field (peer rating 3*0 or ^ove 



'in Boms nd ^dsnnnp 1970) j and (3) the percent of faculty vh© ^ 
rerelved deetoratM fT@m the womi prestlgeeus pfogrras (peer ratings 
2*5 or ab0V^)« fable 5.1 gives tlie memsp staadard deviations p and 
intereorcalattms'of these items. Most faaulty ^mbBTv (an average 
871 ehMilstry departments, S6% In history departmmrs^.. rad ^% In 
psyeholo^ depwtiiients) reaelv^jd their highest degree from a "leading 
|^tl*:»ition" Itt their field* These results are oonslstent with the 
vary strong tffltphasl^o^ the tralalng of researahMs and aaholar/ 
teaohers in the top prestige progrttas^ discussed In fhapter 3, and 
with the tendency' for graduates ^ef top prestige programs to be 
employed by Ph*D. -granting universities (see Chapter 10). The 
jiaMures are so, highly correlated as to be practically Inter* 
^ange^le. . .. 

Both students and faculty TOmberm within each program were 
(Iven an opportunity to evaluate the scholarship and research 
aMllty of the current faculty. Though studmt jtidgments that 
faculty knowledge and scholarly ability are "eKcellent" vary some- 
what with the prestige of the program' faculty ,.fiore th^ 30% q€ 
the stjLidents In almost all programs Judged the faculty to be 
"excellent" or "good%" Faculty members viewed the academic, ability 
©f their colieague4. jomewhat more' critically , and In ways that are 
more conslsten^s^lth exf emal peer rating©. Items reflecting faculty 
and student ratlngi^of the faculty's scholarship pre Included In ^he 
scales to represent the quality of teaching (discussed next) or the * 
acadraic excellence of a program (see Chapter U) • ' , , 



Table 5.1 , . 

Intereorreiatiofis of Measures of tha Quality of Programi 
fTOm whlto Fiiculcy Received Ph.D»'s 



Sourca of 41i.D« 

im Average peer rating 
of faculty quality 



C 
H 
P 



Col relations 
3 2 3 



Hemn 



3.83 
3,42 



.46 

.38 



2. Percent with peer 
rating 3.0 or above 



C 
H 
P 



87 

•95 
93 



87 

86- 

73 



11 

16 
21 



3. Percent with peer 
rating 3p5 or above 



C 
H 
P 



55 
97 
90 



&0 
89 
81 



71 



19 
22 
21 



Note: Peer ratings Used in nhis analysis ware collected/; 
by ACE lu 1969 • Variables 2 and 3 can be ©htalnad for ^acuity 
In any program from data. published in Roose 4 Anderseni 1970.. , 

^ Chemistry (IW4) , H - History (N*25) , P - Psyehology (N«24) 

^5"Distlnguished* 4^3trong., J^Good* 2sAdequate, l"Marginal., 
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Quellp of Teach tttE . , \ ' 

jntm yallty of teaehing in^an educational program is an 
tetpOTtwi atfpa^t of qt»llty regardless of a pro,^ram o-purpoge. 
Si|tidaut gvalratlonis of indlviiual courses aad Instruetnrs have 
^feoMi fairly well aacspted at the undafgradumte le^ tl. Though 
evaluat^g the teaching ©f d^atoral students la mo'« eomplejfy 
s^es often tue inuttuotlon is relatively unstructured^ ttost 
chalman of chemistry and pBychologj departments in the study 
reported that there were regular depaii^tmental procedures fot the 
evaluation of courses mnd instructors by doctoral .students 
(73% and 71% respectively) and also that^most individual professors 
obtain course evaluations from their doctoral students (64% and 
67% respsetively replying *-alwaya" or "of tan*'), FValiiations of. 
instruction are not quite so comion in history departments ^ with 

, 57% .reporting systematic procadures and 48% reporting fairly f re^^uent 
use of 'course evaluations by Individual faculty memhers. Nevertheless 
the dtpartment chairmen 'a reports suggest that depattments are more 
concernec' about the quality of teaching in^ graduate progrms than 
;bi ten is assumed. 

One way to obtain an index of the general level of the quality of 
tej^ching^ in a progrra would be to average such student assessments of 

^individual courses and faculty members.^ However, there are severe 
Idg^tical limitations to this procedure, particularly if compara- 
biiifcy across programs and dlgclplines Is desired, because of the 
difierant procedures that are used by different universities and 
ev^ 'by different departments within universities. Therefore , the 



gvocedurs salsated far this study was to aak dtudents to make * 
general ratings of the characteristics of their graduate faculty 
mm tout^potnt scale from ''excellent" to "poor." Some of these 
quaatlOQs asked for Judgments about^f acuity preparation courses , 
helpfulness to students, excitement for nw Ideas, teaching methods, 
nd overall quality of teMchlng* Responses to seven such items vera 
conblped to form cicala grnre a^ t o represent the quality ^of teaching r 
experienced by advanced graduate students In each program « Details 
of the student scale Nq«^7, Quality of Teaching , can W found In 
Appendix F. . . * • ■ 

* ^ J ^ ^ 

Table 5.2 lists most of the items in the scale tad the a'irage 
percent of students who rated the faculty "good" or "exc^lent" in 
progri^s grouped according to dlaolpllne and to p#er rating' of the 
graduate- faculty. The moat notable characteristic of these responsef 

4 

is that they do not vary very much from *0ne grouping of programs to 
another, suggesting that there are srae good and some not so^ good 
teact^rs ta be found in programs at all reputational levels. 
S^i^dente have a hi^h opinion of the knwledge possessed by theic - 
fa^Lty mtabers. fl^ey also find them reasonably excited about mew 
ideas im the field. But only half td two-Chlrds of Che students 'In 
most progrms think that the faculty provide very useful critlclsiii 
of their work, help them deal with clvswork^ use appl^opriate ' 
taaehing methods, or in general Are very good teachers. The scale 
score do^pp^ted from these observations will b# used to characterize 
a^progm*i quality of teaching throughout the rematoder iof this 
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ReaearGh Preductlvity . - . % . * 

Ths nost frequently cited indicator of tha research eminence 
6t ladlvldual faeulty mfflbers or departments la their publication 
reQQrd# SeveLdl studies have developed. elaborate systen^ for 
evalttAtlng the quality as well as the quwtlty of publications by 
indexing the quality of Journals in whlqh pubXlcatlons appear or 
by coimting citations to published material (cf«, Colet^ Cole, 
1967; dlamentet 1974| Drew, 1975; Porter a Wolfle* 1975). 
Such procedures ara more acceptable to members of the academic 
community because they seem to take Into account the varied quality 
of professional matarlals that may find their way Into print. 
Uoveverp there are a number of methodological problems with such 
measures and. In addition, they require detailed data^ from 
or about a larrge numblr of Individuals within each discipline. 
There la consldarnble evidence that self ^reported counts of 
publications I despite their obvidus potential for error i are highly . 
related to Index^j bf publication quality in a nimibe^^f fleids 
(Jauch k Aueck, 1975). Ther>ifore^ we adopted facult*' salf-reported 
counts of publications ^nd scholarly presentations mb the measures 
of faculty product Ivlt^l^ t^ls study. 

Haaaures of the most traditional indicator of research produc- 
tivity » faculty publications I were dbtalned by asking faculty 
nmbars to record the number of journal articles and single chaptera 
to bool^p scholarly book reviews, authored books, edited books or 
anthologies 9 md monographs and manuals that they had published In ^ 



thalr mitire profesalanal career and in the last three years (1972-75) 
The avsrage perceatages of faculty members who reported publication 
of various ninibers of documents, grouped by peer rating of the 
quality of the faculty In Mch progrm, are reported In Table 5,3. 
leports of muthored tod edited books and nonograpbs have been 
cooblned* ^ 

Discipline differences In publloation patterns ar^ appar&nt 
from the data in Table 5,3, More thm 90% of chemistry respondents 
reported more than five journal publications in their careers rad 
ai least one Journal article in the last three years. Many have a 
total Or more than 50 journal publications to their credit , ^^d^- 
16 or mpra journal publications in tht last three years. Psycl^lO'* 
gists are not quite so prolific ^ but they appear to fpllow the same 
general pattern. Historians reported publloation of many mor4 
scholarly book reviews and more buuks during their entire careers | 
as well as In the last three ymAtm^ than did professors In 
the other two fields. , 

nie different patterns of publication by discipline probably 
reflect the different traditions and eKpectartions for publication* 
that are aharaeteristic of these fields. Chemists almost always 
collaborate in their research, frequently publish reiults as oi 
of sev^al coauthors s and tend to publish several articles on* 

different aspects of one research effort. Therefore* the average 

r 

ntifiber of publications for faculty members in a ch&nistry departmetit 
is^ almost certain to be higher than the average for most other 
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The pattern of collabo^ ition , coau-h^rship -nd emphasis on 
journal articles rather lihan books is also characteristic of 
psychologists, though the frequency of publication is slightly lo^-er 
than amng chemists* Historians ^ ho^ aver » have a different pattern. 
They tend to work alone rather than in :Qllabora. .on with others, 
and to work on one major schoiariy project at a time, su^j. wi- 

th is project is likely to be a book, mo ugh they r.ay pub lis.; 
individual chapters as journal ^-rticles while the book is in ocess* 

r^^r1^q fhAn rhefnifits or nsvcholo- 
gists. Historians also publish more schola ly book reviews, 
perhaps because there are more books to be reviewed as well as 
because they carry more weight as scholarly contributlcns than do 
book reviews in the other two dlscipliues. 

Faculty members also were asked to report the number of times 
in the past 12 months that they had presented some of their research 
results or other scholarly material as a colloquium speaker, visiting 
lecturer on another campus, or speaker at a professional meeting. 
The average prn^ram frequencies bv discipline are presented as the 
last item in Table 5.3. Again, chemists are most prolific, but 
there are no great differences between ^he Hlsrlplines, Most 
departments averaged a median of 3 t 5 presentations per faculty 
member in the prior 12 uionths. 

The self-reported data on faculty productivity were used to 
construct four measures for use in other analyses* First, the 
number of professional articles, single chapters in books, and 



scholarly booh reviews v^ere combined to f^"^r^ nn«^* count nf journal 
publications that, wouxd be r.^a!- =nably c 'mparable across the three 
disciplines. Then 5 because the total number of publications in a 
facultv member's caree'^ is. In part^ a function of the number of years 
in which sht- r he has been a professional in the field, the totil 
numDer or journai puDiicationa Wis aivxu : dv j..uut?i. ui= _ye^i=s 

since the person had received ihe Pr.D. ^ give . measure '"f the 
avViage nnr il race ol journal publication. l.^e^e i dividual annual 
r^r^R wpre then averaged within oroeramR as a measure of the long-te 
productivity of the faculty. 

The average number of professional articles , single chapters 
in books, and scholarly book reviews in the past three years among 
faculty members witliin a department was adopted as the second index 
of departmenLuL faculty productivity, tC' indicate the faculty'^ 
f^urrent level of performance in this area. 

The third area for the assessment of faculty scholarly 
performance j the publication of books or mondgraphSi posed problems 
because of discipline differences , the relative infrequency of 
publlshinp books, and failure to find any satisfactory way to weight 
book and article product ion so that thev could be combined in any 
dafenAble and acceptable form* Therefore, the percent of^ facultv 
menders in a program who had published one or more books or mono^ 
gra*^hs was adopted as the measure of this variable in this study* 
It Is a measure of the extent to which a prog am hss taculty members 
who uave made a major conl ^^ibutlon to their field through the 
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publication of on^ or more hnr^ks^ racher than <t mea>:>ure of clie 
nimb^r of books tba^ they have publish 

The fourth Index of research productivitv , number of presenta" 
tloiis in the past 12 months * is avt : ige of faculty reportb ^^ithin 

program. 

Scholarly Act ivity and RecosnlEion 

It can be argu 'd th.it publication rates are important, but ^.re xo 
sufficient to repre: .at the research and scholar] v excellence of a 
faculty; journaL ar- icies or ouukb ate unl> Lur. Li^^ ol the xccii. eg 
of faculty research rf^ori and contributions to their profession. 
Currant active Involvement in researc. also 's important , as Is 
contact with colleagues tnrough retereeing articles or editing 
Journals, serving as officers in professional associations, and 
^^T-^ring r^v^f^w rnmrnlrtees and site visit teams. 

One section of the faculty questionnaire asked respondents to 
indicate their CKpGrlence in a wai-ie^ prnfe^Rlonal areas* 

R sponses to five of these iterr^ were combined to form Faculty 
Scale No. 3 on Research Activity (see Appendix F) . Departm^-nL^ 
with high scores on this scale employea more faculty members who 
reported that they had: 

" received an award for outstanding research or scholarl> writing, 
^ — edited a Journal in their field or served as l rnember of a 

Journal editorial board . ~ 

refereed contributed - rticles for a professional journal in 

thair field, ^ 

.}^4 



received an institutional or departmental re^^earch erani; 
in the currenc year, 
-^f- racaivad a nonuniversicv grar t or cont ract Co support their 
resaarch in current vear. 

This scale is used in this re^searr^ t_n represent tb. level of facul 

Other prof© ^0 1 o n a I ^ x r r 1 o n c €~ r *j r r t ^ d ^ * f ? u 1 1 v sc-7^:b e r 3 a r o 
presented in Tab^_.^ j.A, together with their reports of the number o 
davs away from carnpus during the past vear for professional activi- 
ties such as iDeetingSj s )eecheSj or consulting. It is important to 
find 3Ut about such activities when investigating Individual 
prL^rams, but i items did not combine wail to form a scale scora» 
and % they were too numerous to u^e individually in most analyses In 
this study. Therefore^ they will receive only occasional mention 
In the remainder of this report.* 

The interrelationships of these measures can halp us answer 
aevaral questions. Do tacuity who perfofti at a high level in one 
area *tend to perform highly in all areas? Are the pear ratings of 
faculty quality, discussed in the previous chapter as additional 
measures of faculty periormance, related to the mora objective 
measures of scholarly productivity and professional activity? Does 
faeulty resaarch activity complement or detract from teaching 
perforaance? The IntercorrBlatlons of eight measures of faculty 
performance are presented In Table 5^3, together with the means and 

i2S . 
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Parcentages of Faculty Reporting Selected Professions 

Actiyities and Recogr. -wns 
(Average Percentage Answering ifes**) 



1* Have you served on government 
or foundation review comltteesi 
site visit teaffls or national 
advisory councJls in the last 
three yea^a/ U=^6-a; 

2* Hav€ you held a postdoctoral 
lellowahlp? (F -S-b) 

3« Have you received a serious job 
inquiry from another employer 
in the last two ye irs? (F-'S-c) 

4. Have you reeelved an award for 
outstanding teaching? (F-8--d) 

Have 'you received an award for 
outstmdiiig professional 
. praetiee? (F'-a--f) 

6* Have you held office or served 
on boards of a nationai profeB- 
slonal association or organlza=' 
tlon? 

7t Have you held office or served 
an cotDsitteea of state or 
regional professional organiza- 
tions? (F-8-h) # 

8. Days away from camp\is on 
professional activltiris in 
past 12 months 

0=5 days 
f6-15 days 
16-30 days 
31+ davs 



*See footnota, table 5»2 
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Tabxc 5. 5 

Intercorrelations of Maasuras of Faculty Performance 
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Th€ two measures of the prod .c: ion of articles reporttJ in 
Table 5.5 suggest a continuity o leVGl of accivity; faculty members 

4 

who hava produced a lairge number artl - throusliout their carf rs 
also tend to hav:„ produced relatively large numbers articles in the 
la^t three years. A similar conclusion applies to the publication of 
books. However, the smaller cr^ relations between the average annual 
number of articles produced faculty :nd the percentage of 

fr^culty members who produce 'i books nuggest that these two types of 
publication activities are relatively Independent. 

As expected, the number oi r ^ eseiiLatlon^ in tiife pabt 12 munth& 
were mor ^ closely aligned with the production of articles than with 
the production ©f books. The research activity scale correlatoa 
relatlvfily^ highly with all of the productivity measures , as cto peer 
ratings of faculty quality. The quality of teaching scale i however , 
linri ^r#nf 1 V meAqurfeq snTnething aulte different* for the dorrelations 
with almost all of the other performance measures In chemistry and 

very low. These results ,atrongly suggest the relative Indepandence 
of faculty teaching excellence as gerceived by students and the 
self ^reported research involvement of faculty members, 

r 

When Interpreting these results » it Is important to recall two 
things: correlations can be misleading, an d^ one of the goals of 
this project was to find way to evaluate individual programs. To 
see how these points bear on the faculty results ^ consider the 
ralationshlp between the average annual production of journal 
artieles md the peer ratings of faculty quality in history ^ as 

^ ■ ■ 
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shown in Figure 5,1, ^Ihe -^.re plots peer racings of history 
faculty horiEOntaily and averac^e mnual 7 lo iicacions vertically , 
with Individual program.^ represented by randomly assigned letters. 
Lines hav*^ been drawn to indii te the mt iri f each variable - 
Perfect agreement bet"*'een the char act^ristics would be represented 
by 1. diagonal line from the lowe left to upper right corners. 
Although the c^^rrelatlon in Figure 5 . 1 is fairly high (.55) , the 
Ijcatlons of Departments Jj and K provide a good illustration of 
the way in which correlations can be misleading for individual 
program review. Lhe lacuXty or uepartment J* cne mcsc productive of 
allj has one of the lowest ratings, and the faculty of Department C, 
which rias a rating just below that of Department averaged nearly 
twice as many articles per year K, 

Figure 5*2 shows the relationship befween peer^atlngs and the 
average nisnber of articles produced In the last three years In 
psychology. The farulty of Department H, netir the bottom the 
peer ratlfigSj is about as producti e as ^^t of Department M, the 
saaond most-^hlghly--rated department. The faculty of Department I 
was more than twice bb productive as Department K's, yet It was 
rated slightly lower. 

Recently It has been suggested that the rate of faculty publica- 
tion In key journals might be used as an objective measure of depart-* 
mental performance in the sciences » since it has been shown to 
eorrei^Ce highly with the ACE ratings ol quality (Drew & Karpfp 1975) 
Tlie eonelatirns In Table 5,5 suggest that seXf ^reported counts of 
publleatlcms In chemistry might do as wellp Mid they would be mch 
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easier to collect. Fh- :k :■ t i cn^., I p ^rund onlv in 

chemistry, however, ■ ; .-1 ; -i-- ... (wnetn^r 

self-reporned or counced froTi; 1 Lb i iographi^s or ournai lists) 
probab ly are mnch -useful -i ^ s i n 1 I nd i c a t a r ■ f a ua 1 i t v in 

fields outside the hard scicn^^:.. :n fact, the maicr message of 
the Intercor relat ions ^ ^c-ciun these ^igiic naasures or faculty 
performance is that each nne of them contril some unique view 

of a depar tmenu ' s chaiacier aisu uualit>. No c ne measure is 
sufflcj^ent to as jss nhe quaitty u; faculty perforralnce or of t;.e 

Correlates uf Faculty Per f urruantLe 

The measures described in the previous section are concarned 
with three major aspects of faculty performances the quality of 
their work as teachers, their current Involvement with research-^type 
activiciee (such a^ eulLiag pt u^^s^iiuueal juutiials, ^livicwing papers^ 
and dbtalning research grants), and their contribution to new 
knowledge in their tieicls twfQugh publications * in this sectian 
we win loc'; at one measure of faculty performance in each of these 
areas in relation to other departmental descriptors* in orde- to 
identify the kinds of programs that are associated most closely 
with different patterns of faculty behavior. The variables selected 
for closer InveHf Igation are the student scale on the quality of 
t;eaching» the faculty scale on research activity, and the average 
^tiTT^H^r of jnurnal pMhi ir.qti^n' the last three years by a depart- 
raent*s faculty members* iwb last , idlcator was selected from among 
the several productlvitv measures in "able 5.5 because It seemed to 



represent most accurately the current status of a program with 

respact to faculty scholarly contributions to ea^L di&clpline, 

^able 5*6 preaents the correlations of these three meaaures of 

faculty performance wich a variety of other departnientEl descriptors 

These data indicate that: 

" stud^t ratings the quality of teaching in all three 

disciplines arer^rt^ngly related only to student ratings of 
the learning environment (8) and personal satisfaction with 
the education they are receivini (9)* and to some extent with 
tacuity rascarcri invaivcment m psycnoxogy, 

— only In chemistry are the number of publications and the Mioimt 
of research activity both highly related to the research ^has 
of the progr^ (1) * "^sources (3), travel away from COTpui (5)* 
and outside research gr tnta (7). f 

but iii all three disciplines there is a strong relationship 
between the size mf the program (2) and the nuniber of recent 
faculty journal publications (despite the fact that the 
piAllcatlon index has been adjusted for the number ©f faculty 
members in the departraent) # 

— thm rasaarch activity scale cof relates more highly with a 
nLmber of other research-related progrMa des^tptors than 
does the publicatlou measure in all three dtfciplines, 
suggesting that this may be a better single indicator of the 
research perforaance of a department's faculty, 

— progms with faculty mei^ers who are acclvaly ixivolved In 
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L ' le 5.6 

Corrilations of Three * iasureb Faculty Performance 
. <vith Othir Depai-tiental Descriptors 



-^.^ ^ ^ ^.^^ -..^ 

t 


Quality of 
teaching 
scale 

C H P* 


Avg. nmHi 
Jounial pubnSi 
last 3 yrs, 


Research • 
Activity 
Scalg 


Propm Chiractirlstics 


C 


H ^ P 


C H P 


1, frograi raphaali on r§iearch (F-2-m) 


13 -02 61 


34 


21 28 


111 

u] 76 67 


2. Slzil RE ddctersl p'tudents (DP-S) 


22 01 29 


77 


48 61 


73 61 51 


3i FietJity^'Eitid 4diquicy of physical and financiil 
risourciS 


34 00 60 


56 


09 30 


75 66 71 


4t FIculty-ratid scholirly excilleni.'^ « program 


24 13 '2 


68 


27 43 


61 62 69 


$• faculty diys iwiy ffOTi ttipus (F-10)^ 


-16 20 29 


71 


31 11 


58 40 16 


it &Eptrlin£il ytarg iiinc# rioilving d .uorati 
' (F-U) 


17 -22 25 


19 


03 -04 


33 14 51 


f 

7. Ftrcfflt of iaculty with outside research 
iraac 


04 27 46 


67 


31 37 


83 '51 87 


S* Stud^t-tatfd gQVirdDient .'i^r learning 


52 63 5S 


-16 ■ 


-25 15 


02 -49 01 


!• Studat satisfaction vith prograi to meet 
thiir niids (S-Sc-l3) \ 


68 89 77 


27 


16 27 


30 34 56 


0. AluBli sEtrosly good prgpara^ion for 
icholarly resaarch (A-4^a) 

i 


13 25 -19 


60 


46 24 


51 60 43 


L Picc^t of. alimtl eurrantly ingaged In rssiarch 
or resiareh ihd tiiching (A-^IS^ ) 


-07^ 04 02 


22 


06 49 


-25 67 S2 



*Ses footAOtii Table 3,5 



rfe^search and publication produce al nni who say tbat t'' e program 
prepar ^ them "ext. emely wfell" for careers in scholarly cebeaiwh 
(10) .-nd these alumni tend tc take jobs invol ing research (11)* 
(Thm negative correlatiun between f^icu^Lv research activity 
and percent of alunmi in research poEitions in cliemistry 
prn;_. ibly reflects the high p rcent of chemists who report an 
appMed or pi .^ct iti '\.etr major job activity In industry,) 

The correlations bet jen faculty age or experlen' (years since 
receiving th^ doctorate * #6 In Table 5*6) and measures of faculty 
performance suggest a relationship of any consequence only in the 
research r-ctivity of chemists and pb/chologists • M re experienced 
faculty meters were not viewed by their students as better teachers, 
but experience perhaps gives tham more visibility and mora opportunity 
for research activity. 

The table of background characteristics of faculty respondents 
in Appendix E gives the average hours per week reported by faculty 
members in each dlEcipline spent in professional aetlvitlfes such as 
taaching, advising students j research and scholarly wrltlngi and 
program administration. As expected, hours Bpent in teaching are 
negatively related to research productivity and activity, and hour^ 
spent in» research end writing usually demonstrate the opposite 
relationship to these variables* Howover, there Bum many mixed 
relationahlps between faculty repDrts of the ways in which they 
spend their time ind other measures o£ ptogram characLtirlMticii , 
suggesting that time estimates are not very valid indicators of 
program activity. 

13.- 
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; , 5.25 

A parelstwt question in higher education, at the graduate as wejl ^ 
mm ths uadetgra^uate level, is the relationship be^een the research 
effort of fs^uley maimers rad their skill as teachers (efi, Fulton k trov, 
1974) # Must graduate programs emphasiEe faeulty reaeareh md publiea* 
tion in order to provide good learning experlenees and good teaming 
for the ^ptmg aeho'lars who enroll in the prggrans? Ta^le 5.7 reports 
the relatloiifthlp of a program's" faculty research activity seore 
ttid the average number of Journal articles published in the last 
three >ear3 with a variety of progr«tt indicators related to the 
lemrninl envlroiment and good teaehing, ^ Though some of the correla- 
tloiis' have been reported in earlier cables » it seems useful to 
bring thm^ together in this way to examine the question of resear "h. 
vs« teachings 

Before investigating these relationships,, perhaps it would be 
useful to poJjit out the difference between a program's emphasis on 
preparing students for positions as college teachers, as diseussed 
in Chapter 3r and the performance of faculty members as* teachers snd 

ft 

as advisers to student&» The attention given by^ a program to the 
training of researchers » teachers, or practitioners is not necessarily 
related to' * the performance of the faculty members in that program as 
teachers or as researchers. However, the wfelght given to the 
performance measurep^^might vaiy depending upon a program's 
puirposes. , , * ^ 

Of the variables related to good teaching tha Are listed in 
Table 5*7, only altami reports of their di^ser^Ation experiences ^6) 
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Conelatlras of Faculty Reflsarch Activity md Publications^ 
- with Eight Variables Related to Good Teaching ^ 



Research Journal 
Activity Publications 







c 


H 




i ■- 

c 


* 

H 


P 


1. 


T— = 

Studant-rated quality of teaching 


11 


15 


59 


-03 


14 


27 




(S-Sc-#7) 




i 










2. 


StudTOt-rtted anv it oilmen t f oi' learning 
















(S-Sc^#l) 


02 


-49 . 


01 


. -16 


-25 


15 


3. 


Student^rated faculty concern for 












-14 




students (S-Sc-#4) 


«11 




12 


-13 


-21 


4. 


Student assistantship experience 












25 




(S-Sc^#8) 


34 


-13 


66 


43 


-32 


5. 


Student : learned a great deal in 












11 




department (S-l^o) ^ 


18 


42 


07 


• -04 


19 


















6. 


Aluml dissertation experiences 










1 






(A-Sc-#1) 


-41 


.48 


54 


41 


28 


29 


7. 


Faculty^ra^ed hummeneas of envir^- 












25 




mmat (F-l-m) / 


32 


-10 


16 


12 


-15 


8. 


faeulty-^'rated quality of faculty/ 










-11 


25 




student relations <F^2-d) 


33 


-19 

1 


03 


13 



%ee footnote, Table 5.5 
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AM ^mslst^tly positively related to the resemreh sGtlvity aad 
p^lioatloa tmtm of a progrra's faaulty« SatlsfaatlOQ with studeat 
aaalatnMhlp ^^rieneM (4) are positively related to faoulty 
feaMreh efforta di^^atry aad psyahQlogys no doubt reflecttog 
the large naber of raseareh aaslstantshlps In bath of theae fleldat 
Aa no^tad ^earlier t there appears to be some positive relatlon^ilp 
batwaan taaehlag quality md research aetlvlty In departnaata of 
paydiology. But^ In ganeral^ ^he patlltn-^la one of a low or 
nagllglbla ralatlonshlp betveen the two variables # Teaching nd ^ 
reaaarch appear to be relatively Independent dimensional auggeptlng 
that dapartments could be exeellent In ,one area and poor In the 
othar« poor In both^ or, as would be th# Ideal » excellmt In both 
areas. Given the llioltatlons of flnanees. It may be'eaalar for 
faculty m^i^berp who are good researchers to substantially Improve 
their teaching than for those who are good teachers to si&stu&ially 
Inareaae thslr resaarch productivity* In my events rsscareh and 
teaching are relatively Independent dimensions for asseasment i^en 
Invaatlgatlng progrm quality.- 

SiaaaCT and iTOllcatlona 

The quality of a doctoral progrw's faculty probably la the 

s 

single OQat important characteristic associated with the aaaaasmant 
of p^g^aiB quality, as Reflected In the widespread use of peer 
' ratings of the qlaallty of a program's faculty as the Index of program 
quiddity « On closer Investigation, however, we find that there are 
swaral dlff^jj^t upects of faculty training md performance that 
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4 




Mntrifaute tq a total picture of thm quality of the faculty In my 
p«rtletU.tf progrtfis ud that thasa eharaetsrlstlei ara net wall 
MprMWtad by any single measure » So&e of the ujor fadings # * 
are - that t 

» assessmsntd of Che quality of teaching are s^st cloaaly 
related to measures of the iearning anvlroiLment ^ discussed 
In some detail in Chapter 8, and' are relatively Independent of 
the research activity or publication productivity of the faculty, 
research product ivity^ as Measured by the publication of 
Journal articles, book reviews, and books or by the 
presentation of research results at scholarly gatherings , 
varies by dlsclpllna and can be measured in a nuoAsr of 
different way^ with somewhat different Implications for 
program assessment. 

— the research activity scales raflecclng faculty involvement 
with professional publications and flnanelal support for their 
research p relates more consistently with other research 
variables across all three disciplines than does my one 

of the publicatiuns measures » wd may be a better single 
Indicator of faculty research perforaance. ' 

— faculty background characteristics such as highest degree, 
tenure I ymBts of- experience do not denonstrate differences 
between programs that are usefully related to measures of 
progm quality. The degree origin of faculty aeatiersp measured 
by peer ratings of the progrims from which they received their 
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hlitett dsgTMSi is sGOi^hac related to ochsr Masures of progran 
' qiaallty^t hut aervas nalnly to doeunanfc the f aat that a veiry larga 

- proportlm of the faculty mmbers who taaeh doatoral studsnts 
In m^mtj prograii In th# study aarned their hlghpat dagifaes fron . 
a hl^a preatlge^rppr^« * 

Three measuifea of faeulty performaneep In addition to peer 
ratliiga of faculty quality > were seladtsd for particular att^tlon 
to Chla ehapter. They were student-'fMMt'^sallty of teaehlng aa a 
meaaure of this Important part of every^ faculty m^Aer'a; profeaalonal 
reapanaiblllty ; mem nimfaer of Journal articles and book revieifa 
publiahad in the la^ three years to represent a department's current 
research productivity and scholarly contribution i ud .research ^ 
activity as reflected in a set of itms concerned with Joura^ ' 
edltlngp article refera^ngy and university or external financial ^ 
support foir research^ Other it^LSi such as professlon&l recognition 
through honors^ avarda, or offices in professional assoclatlonst 
could be added to the list* However ^ these three seem to capture 
the major elments of faculty perforaance reasonably welli ud each 
addsl something unique to the indarstanding of departmental e%cellmce 
in this area* 



f 



, • * , Chapter 6 

' / 
' ^ Thm ability BXkA Qowmttismnt of tha studsats who nm actractad 

to jradtiate progrras rad who stay to eo^lete dagraas also are 

inportant. alamants In a program achiavamant of^ educational 

axcallenca. Tdeaily» there would be co^rehenaive Information 

about -the aaademie ability of tha students entering aaeh of the 

dapartmants In the study. Howevarp this inforaatlon was much 

motm difficult to obtain than we had txpacted. fewer than half 

gt tha dapartmanta could raport average Graduate ^cord iM^tfations 

tast acoraa for chair fflost recent encaring scudents^ and almost all 

of- tha avarage scores that ware reported were in a %iarrdw range 

batw^ln 600 and 7o6. Also^ some of the averages vara basad on 

tast scores of only a small percentage oj^tha students who startad 

a ptograv In any given year. Therefore, depsrtmant-raportGd QRl 

acona wiire not considered sufficiently available or raltabla to 

iiicluda in our data ualyaast We did nqt ^sk student quaatlonnaira 

rupgndmta to raport their own test s^^s becauaa they ware 

advnMd graduata students and an average of four or five years 

slnaa tafcittg the test saei^d too long for Reliable recall. 

In aa if fort to find soma dthar standard naasura of atudant 

acad^ftlc ability at antrwcep wa gxrainad the average CM scores 

that had ba^ aant to each deparmant in the last year. (Seoras 

are aant to departnents at the raquast of the tast^takera; 



syHwrlM af tMt i cores s€nt to mmoh deparoieat vara avsllable 

^« OHE program.) nie avar^a of saons smt to a dapartnant 
OQ tbt advnMd tast In aaeh flsld was plottad agsinst peer ratios 
Of the 4Mllty of the doctoral faculty In tha aue daparoimt^ The 
plpte indioatad a very high level of eorraspondande batwean tht 
We wriablaa in all three fields. This close ralationshlp betwasn 
*av«raga test scores and progrra quality ratings was foimd^van 
though tha average score ranges ware quite narrow— 626 to 747 on 
tha advaiiead test In che^itryp 505 to 590 on the history test* 
and 530 to 591 on the psychology tef.t* Other rasaareh 
daaonstrated that students apply to graduate scMols where they 
think they have a raaaonable chance of being emitted. These 
rasttlts suggest that high prestige progrou attract students with 
higUa.r tast scores * in addition to probably ateitting students with 
higher .levels of tested prior knowledge In tha field. Howevart 
baaause there was no way to know which of these students had bean 
adalttad or which ones ^rolled in a given progrra* the test score 
averages were not used in other data analyses < 

there werej however , several variables in our surveys which 
bore on students' academic tapaeity. These included i (1) the 
average midergraduate grades of , students in each departMnt ; 
(2) faculty ratings of the scholarship of students | (3) faculty 
ratings of stirfents' effort; (4) faculty ratings of students' 
scholarly covltmenti (5) an average Student CoaDitnant md 
Motivation Scale score based on faculty perceptions | (6) m average 



Stirfmt ^Hmleatl0s jMlt score based m faeulty percsptloQsi and 
(7) th$ pttpwtt^g** of atudants in aaeh dapartmanc wl^ planned 
cuMra in nMarch or aollaga tsaehlng. Means p s^aadarcl deviations, 
aad imtareoryelatlra# of these itras are presented in table 6.1» 

Average imdergraduate grades obviously have a number of 
lisltatioms as measures of the acaderalc qualifications of students, 

partleular, student's widergraduate grades may be affeoted by 
the difficulty of' the ceurses taken and the^ overall standMrds or 
the grading practices of College attended i aveirage of these 
grade^polnt averages may not accurately refloat the studrats' 
relative undergraduate achievement * In addition p these advanced 
doctoral students reported uiiiformly high undergraduate grade 
averages (see discipline means for itm #1 In Table 6.1). 
Therefo|fe» though they correlate highly ^th faculty perceptions 
of the aeadmic ability of their students , the rmge of grades 
la now severely restricted , at least when compaMd with all 
^ atudenta who receive baccalaureate degrees « 

Faculty ratings of students' scholarly commitment^ iaotivatlonp 
rad coMimJlQatlon akllls appeared to ^ffer more |.ramlae. Th^ itmg 
that foraad the basis of these scales were adapted from m uarlli^e 
'atudy o£ faculty ratings o^ 'outatandingly goo'd tad pu^r graduatfe 
atudamUroa a set of "critical Incidents"^ of acadmlc beHavior 
(Kaillyt 1974) • that ai^dy, the Items wpre tested in a 
of d^varaa dapartB^ts, and the results were refined b^ factor 
raaiyats'taatalquaa. The it^s and scales which aurv4vad our own 
Btatlitlcal nalyaaa were as follows i 
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Table 6,1 

Corrslaclons imoni Hsasurss of Studsnt Quality 



Qorrelatlons 







-« — 


1 


2 


3 


* 




o 






1. 




c* . 

H 

V. 










* 




3.30* 
3.30 


.15 

.17 

na 

■ uo 


2, 


Faeul^ ntlngs of 
•Adluatal^ abUlty 
of atudaats (F-2*r) 


c 

H 
P 


68 

69 
44 


- 




■ 


* 


* 


2.52^ 
2.74 

2.93 


.51 

..27 

• 9" 


3. 


Faeidty m%tm^mmkt that 
studenta work hard 
(F-3-a) 


C 
H 
P 


72 
57 
37 


74 
82 
79 


- 


• 






, 2.86^ 
'3.25 
3.26 


.38 
.30 
.30 


4. 


faaidty agreamant that 
atudanta ara eoonlttad 
to purault of acholar- 
ship (f-3-k) 


C 
H 
P 


62 
67 
48 


sa 

80 
87 


78 
87 
79 








2.29^ 

2.74 

MO 


.33 
.35 
.38 


5. 


faculty^i^atad studant 
aeamltmant/aotlvation 
(I^Sa #4) ' 


C 
H 
P 


54 
59 

38 


84 
85 

87 


79 
83 
87 


91 
93 
92 






2.67^ 
2.81 

2.85 


\30 
>20 

.23 
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faculty-^ratad studant 
conimt^ationa skills 
(F-Sc #5) . 


C 
H 

P 


58 
57 
42 


81 
77 
78 


74 
66 
69 


91 
88 
87. 


91 
91 

87 




2.29* 

2.60 

2.64 


.29 
.18 
' .20 


7. 


Pareaht of studanti^ 
with earaar goals in 
^aaaarch ot eollaga 
tu^tfg (S-18) 


C 
H 
P 


-02 ' 

11 
"33 


-13 
08 
16 


-26 
27 
49 


-16 
00 

■ 26 


-35. 
-01 
38 


-11 

-16 

44 


65 
84 
58 


.14 
.09 
.13 



*C - Chtelstry (N-24 programB) ^4 • BEcellent or agc«« ■tronily 



R 



Hlsto^ (N"'25 prograas) 3 - Good ot agt«e vlth^esarvatloM 



P ■ Psyeholo^ .(N^24 programs) 2 - Fair or dlaagrae with reseryatiOM 

*4 - A, 3^1, 2 • C *3 - high l*vel In 50-75% of studmta 

* 2'" high laval ia 25-50% of atudnts 



SttidOTt CwwitBsng and Motivation * In departoants which atmd 

hi^ m this sealep faculty said that studsnts » » % 

Do a Mad deal of tmaaslscisd readliig. « 

^ Bandla aaalgmsnts with cars. 

tarn aot*saally dlstraeted from thslr studies. 
Show mthuslssa for their field In dlgeusslons. 
Work on projeets to eompletlon despite setbacks « 
kirn not dependent on faculty for direction # , 
Rarely fall to complete major assignments on time. 

Stu dent Comunlcatlon Skills . In departments irtileh score high on 
this scale, faculty said that students , , . 
Have thought-provoklng classroom comments. 
Show Imagination in presenting dull topics. 
Offer well-fotmded criticisms of others' work. 
Do not^ present their ideas In a poorly organized way. 

T^lt i.l indicates that the scales are highly correlated with 
one another^ sugpPBting chat faculL m^iiibers dc not make^ distinctions 
between these two student dimensions, or at least not when 
aiaraeterlzlng the department's collection of stirfents rather 
than judging individual students. 

Scholarly career goals of students do not appear to be highly 
nUtad t© their tmdsritaduate gradeB or to faculty Jud^Mts about 
«h«ir perferamci as students. Career goals are diseusaed at 
greater Imgth in a later section of this chapter. 



6.6 

Cpgg«aatM ef StudMt Qualit y 

shQuld aOM m oo surprisa Chat scudantB with good gwdss 
nd hls'^y^Atl^s from their faculty tsnded to b# enralltd In 
IttrgB progrms that mra also prestigious p nsear^-orltfitadi 
afflycBti and vell*aqulppsd. To llly€trat# theses findings ^ som 
•alaetirf eerralatss of undergraduate grades smd the Studant^ 
CMBltaant/lfotivation Seale are shown in Table 6*2# Based on thase 
corralatlons rad others that are not Inbluded in the table « It 'la 
apparaiit that students with higher than average imdargraduata 
grades tended to be found more often in prograias which pay the 
familty well; where the average amount of outside, reaeareh fimda 



are highl where the faculty comm tfbm prestlgiot^ program; ^iiere 
the emphasis Is on preparing researchers p not teachars or practltlonera ^ 
ttd where the students, faculty , and progrma are brlented toward 
scholarship. Similarly , professors tend to rate studenta%lgh In 
con^taant and motivation where the resources are considered goodi 
where students are ha^d wooers and competitors | where the faculty 
spend relatively little time in teaching; where faculty produce mmy 
articles and presentations; where faculty feel that the progm Is 
sEcellent and one which is compatible ^ith their Interests; where ^ 
the procedures for evaluating students are good; where faculty are 
active lA research; and where high ratings of the quall^ of the 



faculty are received from peers « f . 

These correlations arf crnsistent ^Ith the ccmion observation that 
the most able students tend to congregate in the^ "best" progrms# Herep 



6,7 



labia 6*2 

CoYT^tlms iatween Student "Quality'* 

Seleeted Pragrw Characteristics » 



Chemistry history Psyehology 



CoCTt latl0ns of AyeraEe Studant 
pndiirgraduata Grades with f 

Nu]d»@r of daetoral degrees awarded 
^ anaually 69 48 45 



EKLC 



Pereentage ©f faeulty fron} highly 
rated dtpartsents 



47 59 40 



Studants' peree^tlon of emphasis 

praparlng researchers 53 63 64 



StudAta' perception of emphasis on^ 
praparlfig teaehers 

Studants* perception of euiphasls on 
pMparlng practitioners 

Studeiita* ratings of scholarly 
"^eiliiallenee of progrm 

CorraiatlAiS of Average Student 



^54 -72 03 

^24 -65 -48 

64. 42 56 



Cc"^tawte/Motivatlcm Scale Score with ; \ 

Faeulty ratings of adequacy of resources 71 44 68 



faeulty vlaw that students have strong 
Mnaa of competition 



81 32 82 



Faculty aalf^ratings of program^ 

•ehdlarly ^cellance 86 ^ 85 93 

raculty rMMrch activity Icale 54 35 * 62 

Paar ratings of quality of faculty 59 56 66 



'4? 



M la wmmi mthmt areas in this study , It appears that the affluent 
and p^Mtlgiima progrws are raost favored. They attract the most 
Ale eti^eatsi vho iA turii have acaess to the best equlpaemt ud the 
MMt highly regarded faeultyi asid who also have the first dirace for 
posltlras In prestigious colleges and research agencies* 

Othftr Stydent-Related Indicators of Wogrm Quality 

Stfveral other measures of the characteristics and progress of 
stud^ts might be considered when assessing program quality. In 
addition to those presented In the previous sections. FrobAly 
the most obvious are assessments of Imwledge or skill a^ the end 



of the course of study, perhaps represented by performance on 

qualifying exminatlong or dissertations » HoveYerp^ though i^th 

I 

of these means of assesment are used by most doctoral pro^ns, 
there 1^ no easy way to compare the JudgMSts made about students 
ijR one progm with those made about students In another progrw* 
"nerefore, performance on qualifying eKams and dlssertatlgns were 
. not includ€d as variables In this study ^ ^ey might, however, 
be useful foci for further research on the cooperative development 
of performance criteria. In particular, the puel of graduate deans 
encouraged the us^ gf^exteinal Judges to evaluate the escellence or 
unlquiineas of dlsserft^ations that had been submitted by students in 
various programs in recent fears. 

The possibility of developing a standard crlterlont such as 
a tMt of luiowledge in a particular discipline, was suggested, to ^ 
the panel of graduate deans that was degcrlbed In Cha'pter 1« 
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Vmmnt^ si^gMtlan was not endorsed by them, perhaps because 

%hmy tonsw tht many probleifls th^f f^irinir? ^nrn tnr^r^^H in 
tryl^ te gst AgrMMnt abmt the contents of such a test at the 
Aoatotrntm Xwrnl in mlmowfi any field. The deaps did, however, 
give nodsrste endorsement to al™ni evaluations of their gains in 
knofflttdgm and skill as a result of attending a particular program. 
Therefore, the alusinl questlonnaiie contained several items which 
Mked recent aluaml to evaluate various graduate school exp^itl ^^nces 
jjl relation t© their present careers. These responses are discussed 
la Chapter 10 and reported in Table 10-6. 

In addition to meaiures of stucie^t academic ability and 
periarmaace, the quality o2 a doctoral prograifl might be considered 
to be reflected In Its ability to attract large nOTbers of r' ±u 
appllaanrs (admissions selectivity), its record of dropouts or 
percentage of entering studenr.s who complete the doctorate (hoidiiig 
p0wer)'t or the length of time taken by students to complerR ih^ir 
degrees ^efflcl^nQy of d^greti piudu4^Llyi4> * ^^^^^ ^^^^ ^^.^^^^^ - 
the Departtantal Profile to estimate the status of a program in 
^ eaeh of these areas ^ 

A progr^'s admissions selectivity was aasesscd by comparing 
the ttittber of new doctoral studehts enrolled in a recert year with 
the n\£EAer who were offered admission , and the n™ber v^ho applied 
for ^BlssiOAi in the same year. In chemistry and history, "he 
ratio* of aAftltted to^applled students ranged fron^ about 20% i i 

nearly 1001, with the median program admiuting about 55% of tho e 
ft 



who applied** The Irn yield of zhese uw^- fiol^^s, lj ''atio 
enrolled to admitted students , was ibou t ^2"- , Thi' percent. age of 
applicants who were admit::ed was much lower in psy^noiogy, ranRing - 
frum 03% to 21%, with a mesdi.tn u.f OSS, -ht: ..aw.^^.-. vi^i^ wa^ 3ii^,.;tly 
hlghar than in the other fijldss cr abo 't 65^^. 

The percentage of applicants who were admitted wa^ use^ as a 
selectivity index in some of the analyses reported Ln other chapters 
of this report J but it did ot prove to be very h^gl.j^y related to 
any of the other mea;-ures. Some of the inadequacy of this measure 
undoubtedly is due to dif f erent^j|ecord keeping ari l adinissions 
procedures on the various caiTipuses, ^suiting in data that may not 
■a=L^ay?5 - < ^tif * - -'i -^-.t^A > l i.<'i) , uiic-.'rtr w^ui-u jp^cit L'>^ ^.^ 
'number of factors other t' m quality that affect the niOTber of 
applications received by a givan department — ^for example, geographic 
location* Therefore, tii^ugh a-imissions Belectivihy qp^Tng at fltRt 
glance to be a potential indicator of program quality, further 
invastigation reveala that there are botri procedural and conceptual 
weaknasses Chat restrict its usefulness* 

Concerning a program* s holding powers department chairmen were 
a- Aed to estimate the percentage of entering doctoral students who 
c ,antually compl-'^te th^- degree in ^heir departments. Their ' Stimate^ 
ranged from 25 or 30% to 90 or 95%; the average completion rate was 
repuLted to abuut 70 to 80% in ^liimistrv a..d psychology and 
60 to 65% in history. The data do not provide an explanation for 
the discipline differences, except pt^rhaps one suggested by the 



relaticnship between rate of c^-nple icn and ^ ercentn^e f :u..dnts 
with some form of financial assis trance . All programs in chv^xstry 
and psychology leported son fornj or linancia^ assistance to ^ore 
thari half of their doctcrrl SLulant.3; the median departr.nnt in 
both fields providfed assistance to more thin 904 ot its udenMs* 
In contrast, histc-^v dapartir,^!, reported a median of 56% of the 
doctoral Ltude-^*=e t.7ith financial aFSistance, A larger prrportlon 
of students withoui; financi. I .assistance may account in part for 
the lower rate ot degreQ cDn^ietion in history, 

reasonably good " jducated guesses", even if not always based on hard 
Hr^fq, invft^t-^ -upd fhp relationshiD between the holding? i iwer 
of departments and a few otiier meas ires of their scholarly 
excellence and learning environment. In history and psychology, 
th^re were low negative correlations between the estimated tSte 
of degree completion and peer ratings of the quality of the 
graduate faculty (^,19 and -.Si); a low positive relationship (.41) 
between these characteristics was founa ^monr^ chemistry programs 
In history and psychology, rate of degree completion appears to be 
positively influenced by environmentrl factors such as faculty 
concern for students and n h..mane atmosphere in the departmant 
(correlations on the order of *25 to ,30); this relationship was 
zero or negative in ^chemistry . These results must be interpreted 
with crutioDi however, because of the uncertain quality of the 
degree coupletion estimates* 



The efficiency of degree production vichin a ^eparuniGnt, or ' _ 
length of time taken by most studenc- to complece the aegree , is 
icnown to vary consideribly . rom field tj field and to affected 
by f actQiPB such as financLal aHsi&cance and lauuieatwry v&. library 
or clinical resaarch. Eighteen to 20 departments in each field 
indicated tl =s average nmnber of vaars required by their studenf^ to 
complece doctoral degrees in recent years by area of specials ^^atlon 
Almosi all of the ^nemistry and psychology departments reported 
4 to 5 years, usually with an additional year or two required by 
students in clinical psychology* History departments were more 
variable; about half of th^am reported 4 to 6 years for all of their 
studencs, .t^lq ine otnKir nnii r^putL^f^ 'hml*^ o ul versi - ii-^r 
students in some areas of specialization. The similarity of 
responses across programs in a given discipline suggests that a 
simple and direct niea^urc of thic program characteristic Is not 
vety useful for comparison across programs. There is no doubt ^ 
however, that it is an area of interest when evaluating a particular 
program, or in research tbit plans to consider several progrm 
charactet Isl ^= cs s imultaneouHly . 

Student Emp ' oytnent Goals 

Other indicators endorsed by the p^nel of graduate deans In the 
area of stuuent achievem^nLs , knowledga, and skill Included measures 
such as p^centrf<e of recant graduates who obtained employme^^t 
directly related to tiieir fi. d of specialisation, percentage of 
recent graduates offercid postdoctoral fellowships, md percentage 



o£ graduates emplo; by doctorat^ = a-a -dinp uni'.-}rsitiBS . Quastions 
along these linfes were included in tn^ alumni c ues c icnnai re and arfe 
reported in Chapter 10, Table 10.:. 

A related question, o£ cuurrit. Is lUe carc^er goals of doctora^ 
students wuo expect to recar^ their degrees within a year or two* 
Table 6 3 shows sf^udent roports t'le kinds of positions they hopeu 
CO hold on completion of graduate school, with programs gruui -d 
according to 'ngh, medium, or low p..t r rating of the quality of the 
graduate faculty within each discipline. The most f :riking 
ohservation from ^ he table is the disciplinary differences in 
career direwLion, A majority of the chemists hopavi Co ohtain 

histortani. aimed almost exclusively to^^ard collage teaching or 
university research and teaching; and a majority of th psychologists 
huped to find positions in universities or in clinical practice. 
There were almost no differences in the career goals reported by 
students at programs with differing reputational ratings, except 
that generally smaller percentages of students at the lowe6t-rated 
progrms aspired to positions in reseaich and teaching at a 
university. 

Wlien we compare these career goals with alumni reports of 
their first positions after the deg^^ee, or with the alumni's current 
positions (see Table 10,2 as well as Table G,3), we see that post- 
doctoral fellowships were preferred by about one-third of the 
chemistry students and accounted for about haxf of the first 



Table 6.3 



Student EmployTnen^ Goal^ Tpon Complating the Dgctorate 
(averages of program percentages) 



Chemistry History Psychology 

H M H M H M 



1 Postdoccoral fellowship 35 22 30 3 2 3 12 8 8 

2* Pre-college teaching or 

aaminiscration uuu ^lu ^ u u 

3* Teaching in Junior college ^21 236 **0 

4, Teaching in a four^year 

college or university 11 18 14 41 45 52 9 10 10 

5* University research and 

teaching 16 17 13 46 39 25 49 40 29 

6# College or university 

achfliniatration 100 113 **o 

7. Research in industry or 

nonprofit organisation 32 34 39 112 6 10 5 

8# S^lf-employed professional 

practice **1 001 3 37 

9. Professional practice with a 

clinic, hospital, or agency 13 3 * a q 15 21 36 

10. Executive positipn (adminis'^ 
trstor, curator, etc.) in 
government^ or other 

nonacademic organisation 1 1 U 1 2 4 2 2 5 

11. Other .321 442 4 1 



■'^See footnote, .Table 5.2 



*Lees than 
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programs* Preference f '^r research in ^ndugtry or with a nonprofit 
orgmizatiQn appears to be realistic, since about 20% of the 
chemistry graduates were employed in these kinds of jobs initially 
and more than 40% are currently in these k^Vnds of positions* Only 
an average of about Ib/^ of the chemistry Ph.D. students aspired to 
employment in a Ph.D, university, but even this level is probably ^ 
higher than warranted bv opportunity, based on th-:^ employment 
profile of alumni* 

Very few students in history and psychriogy expected to gc 
intGk. postdoctoral fellowships, but about 2 out of 5 from eaun 
progrm hoped for Jobs at Ph.D, universities. This goal i-^ related 
to reputational level of the program-^almost half of the students 
at top prestige programs, compared to about a qi ^rter of the student 
at low rated programs^-but , even so, the levels appear to be 
unrealistic in relation to the first jobs or c rrent jobs of recent 
gfAduats^ froin^thc -a^c programs. Many >-tsf riri in^ ^ p^rr Irnlarly 
from programs ^that do not enjoy tip prestige, hoped to teach at a 
four-^year college ur ^ uun^rh.D. university* Almost half cf the 
programs' earlier graduates are currently working in these settings. 
Very few of the doctoral candidates in tUese three disciplines were 
aiming to.ard c^nmunity college Reaching, and very few of the 



graduates wtsre 



©nployed in this setting* 
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Thts chapter has reported or cotnniented upon a nimber of 
indicators that have been suggested to represent the academic 
preparation of students who attend particular graduate departments ^ 
their perforaance as students in the departTOnt, and the level 
of their knowledge and skill when they complete the program. 
Unfortunately 5 most of the measures proved to be not very useful 
largely becau^^ departmental records were inadequate to provide 
reliable iinormation across programs i^ror insLanc^* the avetag^ 
GR£ scores of entering icudents). Craduate school grades ^ the 

i 

most obvious Indicator of student academic p^^fpOTance, is equally 
obvlouslv not apnroDriate for cross^Drogram comparisons. (In fact, 
the graduate school grades reported by these advanced doctoral 
students were uniformly high and with very little varlation—almodC 
all of them 3*5 or above. Therefore , they would not ^be expected 
to sht very systematic relationships even witn wlosely related 
measles of academic accomplishment.) , 

Among the measures attempted in this research , the most useful 
Indicator of student academic ability at entrance appeared to be, 
self-reported undergraduate grades. Faculty ratings of student 
Intellectual conmiltment and motivation appears to have some promise 
as a measure of academic performance among enrolled gtuJents* Of 
the measures attempted here, aximni reports of their emplo^mient 
settings and profLisional involveir nts probably represent the best 
indicators of program uUtcomes* None of these is as satisfactory. 
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as one would wish, however* In particular, more careful thought 
and research are needed to idenciry and develop better indicators 
of the impact of graduate progran - on the intellectual and profes- 



Chapter 7 
QUALITY OF REiOUHCES 

Few would argue with the assertion t' it adequate facilities and 

resourcas are nacessary to conduct first^rat^ activitias of just about 

\ 

any kind. Howavar, in graduate programs the Aead for adequate facil^ 

Ities may be even more important than it is In \other areas. It is 

dif f i'^ult to conceive of historians f ur therir^ acholarahip without 

an aJaquate library at their dlspoF Ij or chemists to carry out 

quality research w- ^ut well^equipped laboratories i Yet, it Is 

quite clear chat the natt ^e and qualj ty of the physical facilities and 

J 

n^h^r re^nnrt^^^ H*^ vqry ron^lHesrnhly af^i^Of?^ grsdiiate programs and 
departments « Where such resources are superior , they obviously make 
the advancement of knowledge more likely to occur p but they by no 
means assure it; where such resources are found wanting ^ genuine 
contributions to new knowledge are possible, but far less likely and 
certainly not very easy* 

In this chapter we shall discuss faculty and graduate atudant 
ratings of the facilities and resources available to their dapartment, 
what kinds of departments are reasonably wall off In this regard and 
whac kinds lass so, to what extent quality of resources appeals to be 
related to the instructional concerns of graduate departments as wall 
as to their research concerns, and finally, to what extant Judgfaents 
about the quality o^^^suurces can be Inferred on the basis of other 
descriptive informatio.i about graduate programs, 

158 



^tud^nt and Faculty Ratings cr Resources 

.oth students and fc^cultv members were asked to rate several types 
of facilities and refeources* They were asked to rate the librar%^ 
holdings* laboratory facilitieis, and computer facilities available to 
th^. In addition^ students were asked to rate the quality of thair 
classrooms and other instructional space j and faculty members were 
asked to appraise the overall adequacy of the physical and financiaj. 
resources . 

From the correlations reported in Table 7^1* we can see that, in 
generals both student and faculty ratings of one specific typ^^ ^f 
resource tend to be higher at places where there is also gr auer 
satisfaction with other specific types of resources* For example, in 
the fields of history ana psychology the departments which receive 
higher ratings fir the ad^iuacy of the library holdings also tend 
to b€ departmente where there is greater satisfaction with the 
laboratory faejllities* the computer facilitias, and the overall 
physical and financial resources. In chettlstry departments, however ^ 
this congruence is less pronounced. Chmistry students' ratings of 
specific resources are Interrelated ^ but the ratings made by the 
faculty in chensistry are less global* Chesslsts' ratings of their 
library holdings lire unrelated to tfeeir ratings of their laboratory 
facilities (r ^ .02)» computer fi^llities (r - .09) or overall 
physical and financial resources (r ^ -.11). In fact^ among chOTista, 
ratings of the overall physical and financial resources are meaningfully 
related only to their ratings of the laboratory facilities (.73). 
We might go so far as to say that faculty membMS* satisfactions with 
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Tabj.e 7.1 

Intercorrelatlons Among Scudent ana ^acuicy Racin^^ uL 
Diffarent Facilities and Resourcea 



Hean^ S . D . 



1, Library haldingi ^ 3' 37 .40 

_ 2.88 .46 



(F^2 -h) H 



p = 2.86 .51 



2. Laboratory facilities 
(F^2-i) 

3. Computar facilitis^ 

4. O^.^erall physical snd 
finanaiml resources 
(F^2^x) 



Studmt ratings oi l 

3. Library holdings 
(S-3-a) 



6, Laboratory facilities 

7. Co^'^itar iacilities 

§. Classrooms and other 
initruetlemal space 
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25 






3.07 


.43 
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39 
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36 
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.89 
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45 




44 


70 


58 
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.57 
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36 
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31 
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39 




3.32 


.35 
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36 


52 


57 


42 
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3.22 


.42 


P 


13 
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36 


16 
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3.36 


.32 
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12 


71 


30 


2,87 


.42 
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44 
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34 


56 
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2.82 


.43 
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14 


69 


33 
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13 


77 


45 


2.93 


,.42 



Noti: Underlined correlations are indices of agreement between student 
and faculty satisfactions with same facilities or resources. 

*C -'chenlstrj', H • History, P - Psychology 

°4 -Excellent, 3 ■ Good, 2 ^ Fair, 1 ^ Poor 
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faullltlfes in hi^;co'r'/ and psvcr. ^^z^^ de^^end on the adequacy of 
rmmour^ms of different kinds, hi': ir chtsmistrv the overall flatis-- 
faction terrined" largely hv th€: J icy of laboratory 

facilities. 

It ,is also worth noting that there is considerable agreameni: 
berwegn the ratingb f students and faculty* The correlations 
between the ratings of these two groups--^the underscored figures 
fa table entries 5* 6* and 7-^--are uniformly high. 

What kinds of departments tend to have better facllltlass at 

least as rated by the faculty? Or, stated anoth r way, what kinds 

of depar "inental characteristics tend to be associated with faculty 

\members* ratings of the departments' physical and financial resources? 

Data pertinprit to this question are presented in Table 7,2* Thes^ 

data indicate that In all three fields faculty members" ratings uf 

physical and financial resourced are associated with the dapartment's 

peer ratings, the rating of programs from which mwibers of the 

faculty obtalhed their doctorate^ the program emphasis on preparing 

researcher? s tho f ECjlty^'p^r'^^^^^ed ^r?br»1flr1y exrel lpnre of trna P'rograni, 

aalarigSi size, and percent faculty holding outside resaareh 

grants (particularly in chcinistry and psychology), ' ^rly* then, 

physical resources and facilities are seen to be -morfc adequate at 

th*^ more prastlgious Institutions which havd established reputations 

for conducting researi h and advancing knowledge. It is worth noting, 

however, that better resources (again, as perceived by faculty) are 

also related to student ^rated quality of teaching and faculty-reported 

% 



Correlations of Faculty Ratings of Overall Physical and Financial 
Resources with Other Departmenral DesGriptors 



Dapartmsntal dascriptors ; 



^ Correlations with faculty 
ratingo of Dverall physical 

snd financiai resourcaa int 

Chemistry History FMychology 



Pear ratings of quality of faculty 



Hmmi peer "^ating of progr^ 
faculty received Ph,D. (F-15) 

Program emphasis on preparing rasearchers 
(F-4-a) 

Faculty aelf-percelved scholarly 
excellance (F-scale #1) 

Ifean salarv-^all department faculty 
(DP-1-6) 

Niffllier FTE graduate faculty (DP-1-2) 

Percen,i. of faculty with outside 
rofiianrr'V) grfltif (F-S) 

Average per-student non-work-related 
support from university (DP-11) 

Student-rated quality of teaching 
(Student scale #7) 

Program enphagls on preparing teachers 
(F-4-a) 
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program eEBphasls on preparing Ceachers, at least in chemistry and 
psychology (saa the last two entries m laljle 7*-)* This relacion- 
ship is important* The fact that quality of resourcas and facilltias 
±1 ralatad to research emphasis and productivity surprise no 

one* but the fnct that it is also ralated to the Instructional program, 
at laagt in chemistry and psychology * is a fact often overlooked. It 
appears that adequate laboratories and libraries are not only essattlal 
to conduct firs^=r£i^e research, but also necessary to teach others 

Problems in Obt aining Accurate Re source ^ nf ornatlon 

Having intr duced specific departmenta asourca indicators Into 
the M ilysea-^e* g, s information regarding specific salaries, financial 
support for graduate students, percert of faculty holding grants from 
outside agencies, and the like—it would seam appropriate at this 
poiac to alert the reader to some of the methodological difficultlea 
Involved in obtaining accurate information on such indlcatorSi along 
with corresponding wamlnga about ^he accuracy of such datd* 

intofmatlon about specif w program resource indlc jcors was 

* _ 

obtained via the Department Profile Form (see Appendix D) . Someone 
in each departtnent—usualiy , but not always, the department chairman-^* 
was askad to provide rairl)^ detailed infarmation about a variety of 
financial vai'lables, enrollment and staffing flpires, faculty research 
activity, and the like. Obtaining complets and accurate informatiosi 
about some of these variables proved to be aKtrsMly difficult aad 
where accurata data were finally ohtalnW, It came at the price of 

^ 6'.? 



SL grMt dmml mors tiMB mnd effort being eKpendetl by someQne in the 
dapafffflent than we had hoped or mpected. Some of the information 
we sought was not available at all» some was available but not in 
^he form in which it was requested (and therefore would have 
required even more time by the departments) » some of it w^s 
available but incomplete i and finally i some of the information ' 
that waj reported appeared to be inaccurate. Even such straight-^ 
forward inforaation as mean ORE scores of entering stud@atSp 
nimber of PTE students and faeultyp percent of applicants f^o were 
accepted^ and so on^ suffered from one or more of these defects. 

Probably the single biggest problem area had to do with 
finances. Including such specific indicators as dollars available 
to students (for research and teaching assidtratshlpsy fellowships^ 
and other for^s of aid) ^ dollars in grant money received by the 
faculty* and the departinent ^ s success record with respect to obtaining 
grants j the latter including such figures as the percent of proposals 
from the department that received fimds over a three^year period, 

Thi^ logistic fact of life has obvious implications about the 
kind of information—and the level of detail required for that 
inf ormation^-that can reasonably be sought in future efforts of 
this kind. It also has Implications fo,r the amount of confidence that 
we can place in the data we were able to gsther, and. in the' 
corresponding correlations and other statistics compute with 
^ese data. We made every effort to check on the accuracy of 
questionable information and to clean up such messlness wherever 
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It was su8peeted« Only items marked on the Department Profile ^orm 
In Appei^ix D were ust.d in data analyses. We think the resulf ing 
progras resourna and facilities Information is reasonably aceurate. 
Wa juspspt that it is not as reliable, however , as informatinn 
received from individual faculty members in each deparcment. 

Quality 3f Rasources Ind Reputatlonal Ratings 

As indicated In Table 7.2, a program's pee^ rating la fairly 
highly f elated to faculty ratings of the physical and financial 
resources In all three disciplines. But just what do these 
correlations mean in terms of leveJ 3 of f acuJ .y satlsfactlor .oth 
within and acrosb disciplines? Thr^e data are sumnariEed in Figure 
7.1, and indicate that; 

(1) With reapect to library holdings, ©virall ratings appear 
to be highest in chemistry dfepartments and slightly 

.lower in histor> and psychology departme7.':s I in terms 
of differences within fljslds, peer rating seems to 
make the biggest difference in history (whex^ & good 
or excellent rating Ls offered by 85% of the 
faculty in departMnts with high peer ratings, as 
compared tu 46% in the low group) and the smallest 
difference In chemistry * ^ 

(2) With respect to laboratory facilities, overall level 
of satisfaction seeti^ to be highest In chemistry and 
lowest (understandably) in history; in terms of 
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Figure 111* SuMary of graduate faculty ratings of deparcmenc facilities 
and reaoutces, by discipline and peer rating. 
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differences within fields, the gap In quality of 
laboratory fsellltles Is greatest In psychology (where 
79% rate the facllltlei as either good or excellent 
in top-rated progrms, eompared to only 412 at 
deparcoents with the lowest peer ratings) 
and. Interestingly enough, appears to differ least 
among chemistry departments (though In the latter 
Instance, the percent reporting the laboratory , 
* facilities to be "eKcellrat" does differ considerably). 

(3) Neither the differences between fields or differences 
within fields b^ peer-rating level are as great for 
computer facilities as they are for library holdings 

) 

and laboratory facilities< 

(4) Faeult/ rat<ngs of overall physical and financial 
resources appear ^o be about the sane for chemistry 
and psychology, but slightly lower In history; ^ 
within-field differences by pee^ rating level seem 
-to be about the mamm in all three disciplines. 

Besides examlntog closely the relationship between a depart- 
fiat's peer rating and the quality of overall physical and flnmclal 
resources, it might also be llliminatlng to study the relatlowhlp 
between peer ratings of faculty quality and a variety of other % 
departtiental Indicators of the quality of facilities and resources* 
These data are presented in Table 7.3. Overall* they make It 
clearer why faculty menbers' ratings of the adequacy of the physical 
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Tabla 7.3 

'CorralitlMs of Five isle^tsd Resource and Faellltles Vsrlablss^ vlth. 



Peer Ratlngi of 


OuaMty of 


Faeulty 






Correlatlma with peer rating iikt 




Ghralst^ 


History 


Faydiology 


Salary of full professors 

(DP-i-b) 


78 


54 

• 


48 


Salary for all faculty In 
deparMent (DP-l*b) 


48 


57 


50 


Percmt faculty with outside 
resMrfh ^ant (f-8-1) 


96 


03 

* 


75 


Average dollars (per faculty) 
external s^port for research 


77 


. -07 


81 


Slsei Himber of FTE doctoral 
students (DP-5-e) 


84 


68 


60 



% 
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«ad ftoaa^lAl resources mre confilstantly hlghtr at thbsa prdgrara 
wtth hlgb peer rattogs« DepartMTiti with higher peer ratings 
en gleo ones which heye higher salaries (for both professors 
and all faculty) p a hl^er percwt of facility with outside gratita 
(botti resMrch and tralntog grants ^ particularly In dhemlstry and 
psychology) p and nore grant money per faculty nember frpm external 
sources (though not In history). 

Quality of Reioureee and Faculty Performance 

The data prasanted in Tables 7.1, 2, and 3 and Figure 7*1 lead 
to a better understanding of the natug^ of th^ quality of raaources 
dimension* By examining the levels of faculty Satisfaction wltji 
th3 vaUous regources and faculties, as well as the kinds of 
departmental characteristics that tend to be associated with quality 
of resources and facilities, we now have a better understanding af 
the resources dimension and how it operates across disciplines » 

But a very Important quet tion remains; How Important are 
such resources and facilities to faculty performance? Or* to put 
the question another way, what evidenee Is there ttat faculty 
performance Is associated with the quality of resources? Data 
relevant ^thie quMtion are given In Table 7*4. 

The research and scholarship measures Includi research or 
other scholarly articles nublished in the last three years,, i u^ks 
published during one's *entire career, presentations pt col^oqula 
ud ^ther scholarly meetings during the last year, and the ;ai::ulty 
Research Activity scale, which includes such activities as Journal 
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Table 7.4 



, CenalALiens of Faculty Ratings of Overall Physical and 
financial Resourcea with Six Perfotwance MeasuY-es 



Correlations with ratlags of 
qualley of facilities and 
resovrces In: 

Chemistry HlBtory Psychology 



Rgjijiareh 'and Srholarshlp 
PVrfoinMncai Indicators : 



^eieles in last three 

yaars (F*9-b) 36 

Bookft* entire career (F-9-a) 34 

Pleli(ir'Ct*t 1 JOL : It^c year 

(f-jt.4> ' 44 

i ..■etW'-h I;; t lvlty 



09 
26 



66 



30 
38 



71 



Ins V; 'j>'" ^ 



il Performance 



Stuu^t T"a;lng of curilculim 
(S-SKaj.e #5^ 40 

SturVat-rated qua] \ty of 

teaching (S-sca.l- ,.7) 34 



00 



00 



60 



60 



170 
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edt oriKilps, Joutoal article *re£ereeing, and research or writing 
mmv^M (for dfttalls, see Appendiar dT- The instructional indltators 
iM;.,adii fltudmt ratings of the currleulinB and studrnt ratings of 
th'i quali'ty of teaching, .In general, ther« aie fairly consistent 
,..'Sitlve relationships between the various productivity indicators 
, :d faculty ratings of the overall physical Md financial resources. 
SpAcifflcallyt 

, . . faculty fatings of a department's overall physical wad financial 
res-., fees are positively related to all four research «id 
rruolarly performance indicators to all three disciplines. 

. . . iMGulty rattogs of the overall physical and financial resources 
are also poslf Ively related to both instructional perfornance 
Indicators to chemistry mid psychology, b it not In histjry. 

. . in general, the quality of a department's resources appears 
to be somewhat more important for the conduct of Research 
and scholarship thai for Instruction, but the need for good 
facilities and resources nevertheless exists f. both kinds • 
of activities.. 

... in general. It \rould appear that adequate, physical and 

financial resowces are less crucial for either scholarship 
or instruction in history departments than in alther chemistry 
or psychology; the correlations of ratings of faleiUtles 
and resources with scholarship Indicators are positive to 
history, but lower than for chemistry or psychology, and 
the ratings of resources and facilities are not even related 
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# 

to thm two Instfuotional^perfon^nee indicators In history 
depavMints • ^ 
• . « thm sjngle reMareh and scholarly productivity Indleator whleh 

ia momt atrcmgly oorralatsd i^th faculty ratings of tha physical 
aad financial rasources Is tha daparCnantal raaaareh activity 
IndaiCi whara tha carralatlons are *75» .66^ rad mJl In 
chanlstryi hlstoryi and psychology raapactlvaly. 

Stmoary and Conclusions 
The quali^ty of a department -s physical rasourcas and facllltlaa 
can ba assessed In two ways. One Is to obtain datallad facts about 
the specific facility or resource In question. This al^t be dona 
by cotmttog the holdings of relevant books in the ilbrary» calculating 
dollars available for research, perhaps musuring the square f oot^age 
of laboratory space and computing a space to user ratiOp and so on« 
A second way is to seek faculty and student opinion about the 
adequacy of the facilities* In this study soma of both procedures 
were used* Sttment and faculty ratings of the adeqimcy of certain 
facllltias and jresources were obtained » and in addition we acquired 
Infortiation about certain specific resources of each program via 
the department profile form (see Appendto D) « Our expcrlancas in 
attesting to collect BomB of this Inf ormationp our discussions with 
various program chairpersons or soto other program representative, 
itfid our analyses of the data as presented in the preceding pages of 
this chapter, lead us to the following conclusions about the 
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prMtlcablllty and potential utility of alternative procedures for 
MaMSing the adequacy of program resources and facilities i 

1) Scudent and faculty ratings of the adequacy of the reaourcee 
aire easy to collects are rellablei and are related in 
es^ected ways to a n^ber of departviMtal descriptors. 

2) Faculty ratings of the adequacy of departmental resources 
are related to faculty research and scholarship perform 
nance In all three disciplines^ and to several Indicators 
of the quality of instruction In chemistry and psychology, 

3) Obtaining cartain detailed Information^ from prograiiB about 

J 

their facilities and resources presets serious logistic 
difficulties; though accurate and complete detailed 
information of some kinds can protably be obtained, 
gathering It will almoit certainly require a great deal 
of time and effort at most institutions , aod even at 
thatp It is doubtful f:hat triistworthy Information about 
certain specific resource variables (a#g.p 
of laboratory space or a dolld<" value afiached to satr^) 
war4 ever be Dbtalned across a large numb^^ of programs, 

4) In view of the difficulties Just discussed» it would jeem 
that quality oi resources and facilities might adequately be 
assessed through faculty ratings rather than through 

the e^enslvet time-cons wing » and perhaps uwellable 
method of attempting to obtain 8peelflc» standardised 
neasurte of all relevant resources and facilities. 
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ChaptM 8 

ALITY OF raE miVIROIMENT 

Uheii dlseussad' In ^onnsctloii with graduAte progruSp the coti- 
cmpt of quAllty hai been ustt&lly thou^t of In fairly narreW t&ns« 
It Is ebarta^latically difinsd in ways that s^haslia tha dlnenslons 
of salltetlvlty and prodaQtlvlty, ar» In eaonomlsts' tanis»| faeul'ty 
and studant '•input" and rasaarGh "eutput," Vlwad In this way, tha 
"bast" progrms ara those that recruit tha moat promising /faculty and 

- i 

atudants, provlda adaquata rasources^ and produca a larg^ ntMbar of 
highly ^ragard ad scholarly contrlbutioni to tha discipline. 

For both studants and faculty , howavar, tha dinaM^n of qual- 
ity that aama to ba genarally ignorad is ona lAich mig^t ba ra- , 
gardad as tha procaas dlmenslon'^*that ist what is tha natura and Qual 
ity of tha axparianca of baing there? With raspact to studants , how 
such was laarnadp what sorts of attitudaa toward tha c^sclplina wara 
convayad» was it axclting or boring, were tha faculty ;accassibla and 
support Iva or unavailable and disintarasted? With re^lipact to the 
faculty, what was the quality of interpersonal relations, ware new 
Ideas supported or ridiculed, was the climate nurtur^nt or anxiety^ 
producing, and so on. ^ 

In her national survey, Clark (1973) foimd that these process 
dascrlptors are regarded as important by a large nipabar of the coun^ 
try* a graduate deans » Clearly, such dimensions hava a direct bearing 
on tha overall mrala or esprit of a department , thereby affecting 
such imiortant md basle character is tlc^ as how plaasant it Is to 
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work aod study ther@i th^ extent to which ona's feelings of personal 
•elfiforth are not being ehallenged or doubted » and the degree of. 
rathualan and/or Meltment one gives to' hii or her work or studies. 
Beyond the ^mediate topaet of such features, howfveri It is likely 
that aueh dlj&ensions of a department have a more lasting and permanent 
iapaat on the future attitudes and disposlfions of the students and 
faculty who spend time there. How graduate students are treated may 
well affect How thay^ in years to comei treat future generations of 
graduate students, and the extent to which m^bers of the faculty 
ixnd an enviroiuaent that is colleglal and supportive rather than 
competitive and cutthroat, may have long^ term effects on how they go 
about addressing rasaarch inquiries or other scholarly undertakings* 

Environment Indicator a 

Two different kinds of Information relevant to questions about 
the daparttnenc imvlroruient were gathered. First , from the dfpar Mental 
profile forma departmental characteristics that are obviously related 
to the environment and might even be thought of as strong "shapers** 
of it were identified. Included here were such descriptors as size 
(defined sev^^tfl ways), sex mix imong students» age of the facultyt 
and the llkat We then obtained inf ormatioa^ from both students, and 
faculty, about their perceptions of the .:/iVlrrnment * As the reader 
will see, 'more ^phasls is placed on thiB latter type of inform. *:ion" 
student and faculty perceptions»becau8e it seraed that how people 
actually expefieticed the envirdnment was what is really Important , 

whataver else one might find out about the department deserlptlvely , 

/ ■ 
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Of thm muxy items of perception Infornatlon, analysea revealed 
to bt measurjAg suf f Iclantly ilmllar aspeatg of thm e&vlroome&t so 
thmt thmf could be gnuped together to form enviroi^ant scalM, 
*rhw« wars cifo sueh soales for students*^ an EpvlroMaiit fo^ Lemralng 
■Mltti eoMlsciag of six lt»8» and a faoulty ConaerQ for Sttide^ ta 
•uia» foraad by anvam Items « For the faculty , iiialyses turosd up 
om grouptog of six envlronmsnt Itras %rtilch we termed the Coaoatl* 
bHity of Work Eavlroaaent scale. The geaeral consent , of thase sealM 
was as follows: 

Invlropmeat for Learning (for students) 

1) Did departmental practices create a lot of tension 
In student si ^ 

2) Did graduate students have the feeling that they 
were being ^ed or .esq^lolted by sobers of the 
faculty? 



3) Did the graduate students perceive the depsrtn^nt 
^ as being characterised by mutual ^respect and con- 
cern between students and faculty? 

4) To what extent did graduate students tend to sup^ 
port and help each ether meet the academic draands 
of the program? 

5) To what extent did they support one another In 
their personal lives? 

6) Was compatitlon wong itudents encouraged? 
Faculty 'Concern for Studanti ^ 

To ^at extent were meobers of the graduate faculty i 

1) accessible to gradikte students? 

2) helpful to stUiients in identifying financial aid? 

3) concerned €©r their prof essiona^development? 

4) willing to of^r gaidance and as^iitance in the 
selection o^ courses? 
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5) willing tQ out of their way to hglp s(ud€Qti? 

6) latrnTSSted In the itudents' welfare? 

7) h€lpful to students in finding appropriate Jobs 
after aoapletlon^ of the degree? 

Cg^atlblllty of Work EnvlroMent. (fcr faculty) 

1) To what e3itent were eonfllctlng damands a cduroe 
of pergonal sti^ln? 

2) Was there usually sufflelent tisie available to 
give work the attention it deserved? ' ^ 



3) To what eKtant ware t^e individual faeulty 
bar-s views about graduate 'aducatlon compatible 
with the objeetives and procalures of the depart^ 
ment? . ' ^ , 

4) Would many kedbers ef the department move- to ' 
another university If they had ^ reasonable 
offer? \ 

5) Do peitssrB of th^ faculty have muah opportunity 
to influent departmentar praqtiees and policies? 

6) How well members of t|^^dapartment get along 

with one anothar? ^ ^ ^ 

The spaelfle Item content of these scales, th^ir locaClo^ In the student 
and faculty questionnaires, and their vsi^lous psychometric pTOpertlss 
(e.g^,' Inter^item correlations, item-scale correlations ^' reliability 
estimates) presented in Appendix F« 

In addition to these three scaled, we included Ih bur sna'^i^ses other 
aaparate questionnaire items that dealt with virlous aspects cf the en^ 
^"Ironoent but that did not fit into any of the aYojmBntiona4 a^alaSp 
eithar because of psychometric reasons (i«e.« tKe Item did not corra^ 
late with others in the scale) or their subf? ^ntiva content was net , 
appropriate* 



In examining the data p- ^tainin?; t: ^ bt\^ fa. 
environment tnfor^stl^n ^i^ ,^f^-p.r, - irticulpr atten^^or 

given to facts that were helpful n answaring 'our que tinns: il) W^at 
is the meaning of' the r n.vironmant diFiensions^^that is, what aspects or 
features f^f the student and faculty a^: i^-omnant seem to be defining 
the environment and would therefore • f ■ be important things to 

Know about graduate programs if r^ual > environment is of interest? 
(2) What sorts of program ur departrn^ nt ch. ract.ristics tend be 
associated with particular kir-^ of ewironments, especially 
characteristics that might reasonaLjly ue regarded as infx^.^ncers of 
cerfaln departmental eiivironmt;nt charactJ: ^istics? (3) What can be 
learned about graduate program environments that isn't already apparent 
on the basis of other a Iready^available information about gradu ite 
programs? and (4) Ap a result of the Ir ruu.aticn marshalled in attempting 
to answer the first three questions, what re^.ommendatlons might eecm 
to be reasonable ones to make about whether and how to assess the 
quality of enviromnents in making judgments about program quality in 
graduate education'^ Tr. ^ t^f ^Tnpfi tip rn addrass ourselves to these 
queatic .B , we shall examine the data separ^itely for students and faculty^ 
•sxce^-u in Lli cases whc./" facjlty porccptions of the studan*^ 
environment are Included, 

The Student Environment 

Data pertainine tw the p aduate student environment are presented 
In Tables 8,\, 8.2, and 8.3, and in Figures 8 1 and 8,2 The information 
in Tables S.l and 8,2 are narticularly relevant to the questions posed 
earil« about the meaning of the environmenf. dimension and its .orrelatee 
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Fir^t, it is avident that the two student environment scalt,- are 
talrly hlghl wui related with aaca othar. Inac is to say, aep^irtments 
where there is a hifh fa .^.it, ^ ncarn ^or students also tend to^be ones 
where students perceive the learnirg environment favorably and vice 
versa. Beyc d this, both scales are consistently positively correlated 
with how members of the faculty percel% e the "himanfcess" of che depart- 
ment a^.d the quality of f aculty-^stud-nt relations. And finally, both 
student en^n ronme.-_ sea' b ara negative' '^ correlated with faculty per^ 

^ n i n n ^ n f f^h^- ^ v r n r n f r. n f ^ e n r I s 1 n 1 r * f ,t ^ ' o n ^^-i f e r r , ; n ^ ^ ^ f i f^^ 

dents. That i^, where such exploitation is high, thc: student environ- 

ment, as measu "d jy ^hese scales, is low. Note also that the differ- 
l 

ences among \^ is cipli;4es ca these five eiiVirotment indlcHtors nre gen-- 
erally negligible as evidenced by the very similar means. Only in the 
case of faculty exploitation of students to advance their o\m research 
(variable number five in Table 8,1) is there any sizable difference* 
Iv this case, less explQitation of students is pereeived in history, 
wht re of course the individual nature of research and scholarship (as 
opposed to tne more group or team-oriented rr search in chemistry and 
psychology) is probably a major factor accounting for the difference. 
The lable 8,1 correlations, along with the specific item content 
and the inter-it^m correlations that went Intu the construction of 
• hese two scales in the first plane (again ^ see Appendix t) , lead 
tn the conclusion that the environment dimension measured by these 

scales are hoo^gen^ens (that is, consist of a nisQber of departmental 

/ 

features^ that "belong together'*) and are related iri expected ways with 
faculty pMceptions* 
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Intercorrelat ions Among Se\ eral Indica tora 
S tudent EnviroriTnfent 



Graduate 



Interco: rel atinns : 
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Mean 
______ 
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1. Stud^int scale 1H: Environinen*: 
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for Learning 
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2,81 
2,79 
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2, Student scale /M : Faculty 
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^.onccTn for Students 
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2,71 
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J* This department has humane 
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63 


50 






.^97^ 


,39 


anvlronment as Derceived 


H 


28 


09 






3.08 


.27 


by facu" (i i^m) 
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,41 


4* Quality of faculty /student 
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62 


52 


87 




3,01^ 


,31 


relations as perceived by 


H 


56 


54 


74 




2,92 


,23 


faculty (F-2^d) 


P 


70 


53 


93 




2.91 


,35 


5. Some professors unduly 


C 


-42 


-08 


-60 


-43 




,40 


exploit graduate students ^ as 


H 


-08 


-13 


-59 


-32 


1,49 


,23 


raportad by faculty (F-l^f) 


P 


-43 


-54 


-55 


-48 


2,19 


,31 



= Chamistry, H = His ory, P ^ Psychology 

^4 ^ Excellent, 3 ^ Good, 2 ^ Fair 1 ^ Poor 

c , - 

4 ^ Agree strongly, J ^ Agree with reservations, 2 ^ Dlt ree 

with reservations, 1 ^ Dl' agree crongly 

r 
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How dc t ^.y reiace other depar trripntal inf or-- ition? re 
spfeci:i..^ily , wnat kinds of programs tend to be hir . aad low -^n 
thege dimensions? Th data in Table 3,2 begin to give some clues. 
First* the two var^abl ^'wat correlate niast higbl%' with nhe itu-- 
dant environment scales both hava to do with teaching. Specif ically , 
student-rated quality of teaching and faculty-perceived emphasis on 
tisaching, variables ' and ^ respect i%^*iiy 3n Table 8.2, are both 
positively correlated with both environment dimensions in all three 

4^,. ^ ^u^a^^ ^-Q^r, -.-v. rFDuf^tinnal rnting off^ rvn^rams 

from which departments? 1 faculty received their doctorates, and repu- 
tational ratin of quality of graduate *^aculty~Lhese characteristics 
are e -her not correlated with the student environment scales, or 
In some instances (and occasionally in one or two disciplines) are 
negatively correlated. To put these findings another ws^^. it would 
appear t: at departments that place emphasis on teaci ...g fairly often 
turn out to la J^p^.tr-ncs in which students are generally satisfied' 
with the learning experienre _ and departmantF' which have gained 
national acd " r the quality of their faculty and their record 
of research ttua to be departments in which there is l^es student 
satisfaction with the environment, but the latter association Is less 
flrm*^In all three disciplines, some of the departments with vary 
high reputational ratings are also ones with high scores oa the stu- 
dent enviroruDent scales. 



Table ^.2 



Correlations of T^-i Scuden Environment Sca^ti* with Various 
Da par tmental Descr ipt ors 



Environmant Faculty Concern 
for Learn Ing fo r Students 

Chem . St Psvch CH^m Hist Psvch 
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1* Peai rating of quality of 

graduate faculty -13 02 -15 -39 08 

2. Size: "TE docton students 

^D?-5-2) -24 -46 -04 -^01 -33 15 

3, Faculty/Student ratio 18 34 16 -08 ?3 -15 

4* Peic^nt of students who complete 

degree (DP^^) -34 31 27 OB ^3 26 

5iy S tuaent-^perceived scholarly excel- 
/ ijncfc cr ctoccorai program 

J (Scudant scale #2) 25 36 44 29 49 53 

Faculty'^percelved scholarly excel- 
V lence of doctoral prcgram (Faculty 

^^■wile //I) ' 17 -21 30 15 -14 46 

7. Student-rated qualit> uC L^achiiig 

(Student bc^U j2 63 58 76 81 75 

8s Program empljasls on resiarch (as 

perceived by facu' "y) (F»4) -0^ -42 10 =05 -32 28 

9. ProgFOT emphasis on teaching (as 

perceived by faculty (f-4) 31 49 17 20 58 

10, Percent ^^f faculty with outside 



11, AvOTage fa"!uity publications n 

last three years (F-9) -16 -25 15 -13 -21 -14 

12, Admissions Belectivity (DP-10) -22 -25 -25 00 16 10 ' 

13, Percent of JocCoral students who 
received so'iia form of financial 

afisistance (DP-11) ^04 5. -09 -11 42 14 
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Thftse tendencies are hopefully raade clearer by tha illustrations 
/ in Figures 8.1 and 8,2. In these two figufes, scart piJts depict- 
ing the ualationshlps between deiartment raputatlona?. iHtings (acrass 
the horizontal) and student EnvlrQng.^nt for Learning score?i (on the 
-iticinl) are prepented for chemistry (Flgura ^r) and hiscoTy (Figure 

(^fhese figures are presented for liluF^tratlve purposes. Thus, 
slmij ^-ita for psychuiogy dej^r ^i^^uts * uhough ^viilablo, are not 
pt^BmXiV "r. each ^ gure, 1 tn^s representing the meLns on both 

d£partL^ ■ ^ted by letters. T^us , dv^pa tment^ with ilgh 

^'scores' cp.. ^lional Rating and EnvirDuncnL fov Laaruing wiil 

be^founi m the \ ^ight-hand quadrant, those with low scorea on 
both vntiaoles l . l. s ro^^^r left-hsnd quadrant, and go on. 

If ws exaniine ha chemiFcry department dat*^ in Figure S,l, Boum 
subtleties of the rel^tLonship between thes ^ two dim^nslonfi bacomes clearer 
First, as iudi^iated by the correlatlQn coefficient (-.13)* there ta 
fi very alight tendancy for departments vth higher reputational rat- 
ings to have somewhat lower ^tudent-perceAved envlroM@nt« tor learn- 
ing, at leapt fo- the 24 chemistry departmei* ,n our sample- But 
thl^ tend..icy la very we^k, and it is ^U^t i .at knowing a department'fl 
stitus with r*. r t to Its reputational rating dof§n*t paralt a very 
A-curate prediction of Itr status ^*^in^ l aspect to its envlronaent 
for le fnlng ^r j vice versa. Indeed, of the t^^^'o cheminiry depart- . 
»entB with th*i '^Igb " reputational ratings (departoe«t8 V and U) , 
de^a. ment P has one o£ i e lowee raced envlromienta for leaming, 
whc;.r^'.&s depsrtaen; U hbs jne of the highest. 
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Even where che relat ' ""^r Ip b-i^tween the variables is s*"r mger— =- 

as is the case for the jame two characteristics in history departments 

as pDrtrayed in Figure 6* 2-^-- there are nufflerous cases in which depart- 

mCiiCs " status on the two variables in queftir^^r: do not ronfora to the 

endancy for the total group. Thus, t/iou?!/ fiaere is even stronger 

1 

tendency in history dapartments for these with higher reputational 
ratingi to have lower Environment for Learning scores (as expressed 
by th^. ",48 correlation) ^ we see that departments N, D, and U have 
high reputational racings and high learning envlronffient sccres* where- 
as history departments A, and X are below the mear on bofh 
characteristics * 

ihus s we have learned several things about the nature of the 

environmental characterist Irs can be measured fairly accurately and 
reliably; seconds the quail -> of tue environmant can be measured with 
"validity," In that data ^^bt ^Ined ^Hout the enviroiment correlate 
in expectfc ways with othe^^ information available about departments 
(e,g.i affiphasis on teaching, size); and third, in spite of the 
corralations with other departmental characteristics * the quality 
of the environment is something which cannot be easily determined 
simply by knovlng, for example, that the department is big* or that 
Lt emphasizes researc^^, or that it is ve^ v selective* Though such 
.actors are relr.ted to the environment dimensions ^ the relatlnnHhlps 
are such that the quality of the e/:vironmcnt Is dt rerent from ^hese 
things. 



One )tl\er point about trie hl-^^ c^nvf r c t i£ worth making, 
i 

Thoii^h our u^der tandi .g of the nature of h^ environment leads us 
to fecil confident tha" it functions pretty -iu..h in the same ways 
acios£^ each of the three disciplines in c ir t udy^ the magnitude of 
the reiation?ihl:^s varies. . ^ms, thoMgh l i qi.iai ty of the environ- 
ment is viewed more favorably by ^^te students in those departments 
where teaching emphasized in alj three dis( Iplities, that relaticn*- 
ahip is more pronounred in the history departments. Cc iversely, 
though the tendency for the nuality ^ f the environment to be lower 
In departments that are large and "mphaiize research holds ' r all 
three disciplines, these re^aMonships are again consistently 
stronger in htstory than in chemistry or psychology. We might isay 
that the quality ul uhe ^ladual^ student cnviranment ^ore 
predictable in historv than it Is in the other fields. Unfortunately, 
there is nouhing In our data that enables us to explain th^a phaiiomenon 
and this wouIl not seen to be the proper forum for epeculatlon. But 
these facts do aiert us to the complexities of the environmental 
assessment prob : am. The procedures used in uur study sero to le 
anproprlate for us " ill three fields, but the scrangthfl 
of th^ilr rel flf inn^h^ps wi rh other departmental factors clearly 
cannot ba generalized across fields. 

So that the readf^^ can examine che extent of student endorse^ 
m^^nt of the various iter s that were employed In our ^nalyais of the 
student environment (including items in the two s^jdent envirorimant 
scales), a si ramary of these data, prest 'ed sepacafcely by discipline 
and three reputational--ratings groupings, is given in Table 8.3. 
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Table 8, 3 

De. 'ptors of the Quality of tre Graduate Srudent En ^ iromnenc 
( Av «. ^ ^ ■■ - P r o g r am Percentages ^> 



Cheir ' s t_r y_ His tory Psycholog: 



(Percent answering ''Agree Strongly"') 
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students (F-l-f) 
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Faculty/student relations^ (f-2'-d) 


33 
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14 
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^ercmt indicating statement characterized more than 75 peisent of 
studrats in dapartmra^t. 

b- 

Parcant uswerlng "excellent.* 



■niesi? data enable the rta-ar to get a g -£1 picLure the level 
" ' student ^ i^fa *:lon ^^-^rinus Irn-nnr . ^ elements of the 

anvironrif nt dimensions » cnmpara the?^ levels of satisfacticn wit 
those of students at depart^ants r other reDUtational ratings 
within their own fie.d, and fir ^lly, to compare thv dati acrass 
disciplines. In this last it^dpect, it is worth notice, tnat overill 
levels of student f^atisiac ti )n t h cheir envir mments do not a jear 
to differ grea^^y among tha three disciplinas in the study. 

The Facul ^^y Envlronmerrt 

ra pertaining Co the far icy anvironment ciTl presanted in 
Tables 8.4 md 8.5 and in Flgur 8,2. The data in Tab iS 8,4 and 8.5 

pn^*^iculnr ! t^^i^*=,ii 4r^ rr^v^^T^^ n« ^ ^^tfer understandine Oi. the 
nieanlng of tl 3 faculty environment dimension* 

The Faculty Compatlbll j ecal^ (whose general content was 
described earlier in this ch:.Dter, and v:,uae sp-clfie details are 
available In Appendix ) ±s ositively correlated, for all three 
ai^scipllnes j with such other eavironment indlcu.orq as th^ extent 
to which different scholarly points of view are welcome i the 
extent of faculty involvement in the governance of the department , 
the quality of faculty-student relations in the department, thfe 
"hiOTaneness" of the de* tment, ^^nd both the departmental and 
miiversity-wlde intellectual anvironmei t* On the other hand, the 
^umpatibility scale is negatively correlattd In all three dleclpUnes 
with the extent to which faculty l umbers pe^^ceive the depiirtmnnt to 
he a eollectlon of individual? and a place where little or uo team 



Table 





In tercorrelations Ajnon 




veral 


Lvirnnmental 


1 ndi 


cat or s 






_ — _ 






" cu 


It:- 








. — 




— — _ . — - _ _ 




1 


2 









8 Mean 




1 


CompatibiliCy 'f faculty 














2.81^ 


.30 




work^ anvironm&/;t (F^.culty 


H 












z . 89 


. 18 




scale if 2 J 


? 












2 . 83 


. 22 


2. 


Different personalities 


C 


86 










3.25= 


,38 




and scholarly pd^nts of 


H 


41 










3,35 


,29 




viaw are walcotii (F^l-c) 


P 


50 










3*18 


. 30 


• 


Wide faculty involvement 




78 


SO 


- 






3.10^ 


,6 




in importanf departmental 


H 




64 


- 








♦ ^7 




'eci^ ons (F-l-'e) 


r 












1 07 


41 




Department has huniane 


C 


72 


72 


56 






2.97" 


.39 




environment (F'-l^m) 


H 




55 


51 






'O 








P 


87 


J i 








^ 3 7 
^ - Q / 


» Hi 


5. 


Faculty/ student relations 


C 


53 




3^ 


87 




3,01^ 


.31 




in department (F-2-d) 


H 


27 


33 


13 






2.92 


,23 








^ i_ 




} 


9 J 




2 , 91 


. 35 


6. 


Department really a col- 


C 


-61 


-58 


-fh 


-55 -49 - 




2.55= 


• 53 




lection of individuals ; no 


H 


^27 


-16 




^48 -38 - 




2 . 61 


, 35 




truly team efforts (F-l-n) 


P 


-65 


-17 


-2 = 


-73 -77 - 




2,55 


*43 


7. 


University intellectual 


C 


52 


45 


29 


23 19 -Q2 




2.39^ 


,53 




environment (F-^i-a) 


H 




36 


30 


09 -10 -18 


- 


2.79 








"P 


Jo 


31 


2 


23 06 16 








3. 


Departrnsnr intellectual 


C 


88 


81 


o3 


70 54 -56 


56 


3* 06 


^ "a 




enviroii.aent (F=2-b) 


H 


67 


hi 


48 


52 19 -34 


59 


J. 09 


.SI 






P 


67 


45 


10 


57 53 -35 


65 


3.12 


35 




a 

C ^ Cii^nii^Li. y , H ^ Hi^tur^s 

















^4 ^ Exrallent, 3 ^ Goods 


2 


- ^a 




1 ^ 


Poor . 










^ Agree Strongly, j = 


Agree wiuji 


reservations^ 2 


« Disagree with 





reservations, 1 = Disagree Strongly * 



M 8 



efforts take pia^a, though 'i^ain the eorrexatiL :s are Xowesit tr. 
history dapartments wh€ 'e, as we mentioned previo^^^^lv ^ fhe neture 
of the rf#€ar;h and scholarship activities simply are not well-suited 
^to t^&L efforts in fhe way they are in the other two discipl iiuas . 
In any event , these ^^ner^l relationships eiif onsistent across 5^1 
t irea disciplines and of such a magnitude ar . co^ slstency that they 
giv^e us confidence in the "validity" of the faculty envf amnent 
dlmertsion * 

^ . What oth r departmental characteristics can^ to be associated 
with the quality of the faculty environment? Data relevant Xo this 
qviestion are given in Table B, : where the Facult^ uo apat: bility 
?cale and faculty perceptions of tlie department l^itellectual envtron- 

descriptors* These data indicate that: 

*,.ualike the student environment, the quality of the faculty 

envlronn^enf is po.^ltively correlated with faculty raputatlonai 
ratings, an aoph..jls on researcl.i and a record of research 
productivity in all three disciplines, 

#*,the faculty envlronmant quality is also quite highly related to 
faculty perceptions o£ the scholarly excellence' nf the program j 
a dimension whiun includ s faculty latlngs of student compt 
(for details see Faculty Sea in AppandlxF). 

, * *JudgEienf:s about the faculty en/iroMent are positively correlated 
with faculty sat' factions with the pL>slc# and f inanciar' resouicea 
of the place » as well as th age of members of the department 
(as maaaured by years since receiving the doctorate). 
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Correlations of Tv.^ V:i. :U 
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3.. Repucacional rating of qyality 
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Intellectual 
Environ. (F-2-b) 

C H P- 



5H 71 64 



2* Mean reputational rating 
received doctorate ^F-15) 



56 31 



45 70 bO 



3. Average annual produ* ■ t i v i tv 
of articles (I-y-b) 



ly OB 
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4, Progr^ emphasis on research. 



5, Self-perceived scholarly 
excalience of Ph*D, program 
(Faculty Scale #1) 

6 Faculty ratingB uT au^qu^^ 
of physical and fin^clal 
resources (F^2-x) 
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65 47 38 
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7. Siie^ FTE graduate faculty 
(DP-l-e) 



19 10 02 



8* Percent of faculty with 
tenure CF-^IT) 
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-10 19 -02 



9* Years since receiving doctorate 

(F-14) 24 47 
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C = Chemistry, H - History, P - Psychology, 
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as cr t:fj ;u--,t.-:.: :n , . / i .:. Lr/:;..ry-, 

^Iwn ccmr ired vitn iL-/ u.:l3 periain;^-- £hw qualily o: chc 
stuc.ent environment ciisciifeHeu cj^riiur m cnis cnapter, it is clear 
that the qi^allty of the faculty Gnvironinert is bas^u on different 
features of the envi ronrT;tnL . Wherrz^K favor able ^.t dent percept ionn 
seemed to be charac^erigt : ? of. dopar trrinnts that emphasizCv. teaching, 
favorable facultv perceptions oC the environ_ment are obviously (and 
not surprisingly) associated with an emphasis on regearchj ^Irst-rate 
facilities, and peer racings of tlu:= quality of the faculty. Thus, 
departments witi: favorable climates for the facultv are not always 
aepar tnicnL:^ witn c^v ir'jnr^c:' : zr^at ltq percG^vea r4:voraDiy oy 
students* In fa^. t . it woul.. appear that in some cases ths very 
features of the mvironment that make for a desirable dapartment for 
tAembers of tht: faculty often operate to have a negative effect on the 
quality of the environment for students. Only in-^^^giiQlogy does it 
appfear that student and faculty perceptions of the environment tend 
to "go together/' any even here t\.^ relationship is not strong. 

Correlational data, necessary to explore the ways variables 
relate to others, tell us nothing about the distribution of those 
characteristics among t\^^ Jwpartnidnts in question* So...*^ of this 
information Ib included in Figure B,3^ in which faculty ratings of 
p vririety of environment leatures are summarizeu, and permit a 
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"t-put ^ ional ratint:. F:m" : ■• , . ■ - ^ ■ : ■ p^.. r ■•n.:t ; r^ictf • - 

Mon^ rhe faculr'.- vh- r : a r "r;- r . ■ ^ " = ^t-r. . i rhr t'ir. ^e*: 
bar vranhs in Fl-^ar-i 8,3^ vcr c ^^^L^ that: trie ovi^ralL 1 vel of 
pei^-i-^l relatloHii appear ■ diit^r siighcly among he disciplines 

■ uc dif "ers fairly substantiial ! y within disripline by reputanional 
rating lev^l, at l-.-^^t . ..i;--:ulr- ry ^ii^^. Iiirniuvy di^parLmt^nLa . v an 
aside. It is i n t er*^^ ; 1 1 £^ --e ih.it in a. cav.c: i:;:; f aculf: .- per^unal 
T elat ions') tega'dk-.^ iir^.)? rxcelit^nt hv ^ . . as iO% of the 
faculty. ) 

Xnterprp t .1 1: 1 ons of t lu^ .-^!ir-^^ i^^? /^ -.^,^onr!pd in Figure 8.3 ar-^ 
cnat: , 
* . dii r erent p l&oh. iiti^/j and scholarly points or view ari? almost 

disciplines J and though buch openness is less characteristic 
of departments with lower raputations than highsr gnas, this 
dlfferehce is not greatt 

- i^Tt^^'^^^t ir fl^f^^^^r^ to ?iF»/^fVto^ 1^-otr^ '*j*'lo'** 1? '^o^^Eist^"*^!** 
greater among departmentB at the.lov^and the raputation scale 
in all diaclplineS, with Itaiiinga iu Lhl^ uJ^cwLion h^Lag 
greatest in chemistry df^partmants. 

i 

, , ,a seiise of colleagueshlp (Joint effort) occurs with aoproxtmately 
equal frequency across disciplines; within disciplines, 
reputatidnal rating makes the gtaatesc difference in psychology, 
whms^ 65% of the faculty iri c-op-rat ^^d jrograms agree (strongly or 

■ * 1^6' 



'J 

-.1;: at ctepart.r?ent v;:h r:,. I-wm^: j l a c : ons , 

, , , n eve I opm^nc cf iunior i- i/. i'-- r.^:=-^ :ri:-":uen!: 

chemisrry depar cn^jnts , lea-t irequent in [^^ycnology depar trient s ; 
wfuhin Jis-lrnina^, su..h dcvQl-D-iont --r^^ likely tn nrrur 

highly rated prQgrs^'S in cham'Stry and psychology, b " 
reputaL nnal ratir i vos t.::-^aeri to make irr- n difference in 
history. 

SiriTiar^ nnd Conclusions 

lae .,ud.;L. li.. :ac;:lt> nvlrcnnent. a dinsuBl- 

cf graduate cdacation liv.r ls usually ignored, was endorsed by a 

beine a cliaract. iscic that is 
important to considL -at icns ol progia^ quality. In this study we 
measured student and faculty perceptions of cheir departmental 
environments by means of questionnaires thac sought information about 
faculty accessibility, interest s tudenr welfare , extant te wnich 
competU^on among stuaentG is encoura^^d, qu^:.ity of facuity interpersonal 
relations, extent of faculty opportunitv to influence departmental 
decisions, and the like. nese data ' indicate chat: . ' 

,,,^he qttM'lity of the student envircnment seeiad to be largely a 
"humaneness'* dimen Ion, in which students perceive a receptive 
faculty who are inter^sttd in t:..^ students' welfare and give 
attention to teaching. In chemistry and psychology, t^his type 
of environment does not appear to be related to such traditional 
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indicators of depar tm^:nt M. nu^iiitv :i- r.-r'^ ^ t or: , 1 r/i : In ;: , 
publicat ions j 'ind th^ like. In vtber wards, humane erv ironmun cs 
are f our i approximately - often amonH; o^^^ t igioi: ^ dep/;r tm^^nts ih 
lesser known ones in ihese fieldj^. In hisiory denartmen t s , [;ow-= 
ever, fa "^raM^ --tu^.-rw. ^ ; , onni^-u ^ l^^s^ likely -.^ fnunfJ 

in departments with strong reputations and a heavy research 
orientation. 

the quality of the faculty cn^' Ir j^nment , on the other nand , is 
..3S0ciated with repiitational raninp^, : rogran^ emphasis on research, 
quality o"^ the facilities, and the like, a clear indication that 
■ ..^ - ^ s important to tne i acuity is not alwavs what is important 
to graduate snudents; the faculty environment may nr " exactly 

*t!ie measures of environment used in this study appear to be 
reliable and "val d/' and tw pro\ de useful information, about 
itudent and faculty environments in all three dlsclpHnfH. 



Trie prt: eding chapters have baen ccnce ruitd primarily with chfc 
human nspccCs of docroral d, zrec prc;^rarns ^nJ wiLh t.lie adt-quacy of the 
programs'^ physical rasource? , For soma people, this is sufficient to 

assure an educatiefial cxperienca; bring ^ooC mindp together, thay 
Bay j atid qualU:y education will take place, But doctrral pror^rams 
at mu.=?t univers i ties woula iiot exist without much more structure 
tnan this simple iormula suggests. What about courses j requiremer ts ^ 
examinations, supervised resaarch, anii supe: visea work experieri 
"Hiese aspects of a progrim express its purposes , make up a ' irge 

mrchanlsm for certifying those wlio ■ 11 awErded a dagree* No 

assessment of program quality would be complete thout considering 

Its formal structure of learning exi ^rlences and procedures. 

Thbugh currlculu^is vary within a discipline s wall ao betwean 

disciplines i it lb possible Co ask some general questions about 

my program's acatiemic offerings and pToceduras. Do the academ ' ^ 

offerings J whatever !:hey may be, seem to meet the naeds of r\ > 

studentfi who are* enrolled? Do students and faculty think the 

raqulremants are appropriafe? Arp renulrementR fairly ndmlnistered? 

* 

Is stU'' t academic work evaluated fairly and appropriately? Would 
Etu'iynts rDComiend the program tc othcrc with :jimllar interests and 
ablHties? Such questions conce n hi integrity and fairness of a 



they ask iK:uit: ih^^ , , H : ■ %:i:.ia . ^i-'gr.n, r^uarli c^h: 

of zh - facuitv^ri ;r H.B^Lrr. ..r : :: r ^ruu.n-s' ^c-: sccr^s. ir^^/s, 
thW focus or aspects n- -r^^r^-^ !:iwit: are .litercv.L irom area^ ret^erted 

i;..^ude procesf as w^:dl ib ^^choiarly input and output, 

TG^crvti i'^^^ ^ n/f- exp^esVstfd tibuut: Lhe u^^a of 

average par' U i ) iut rdLiuk^^ Lo r-iw^esant trie quality of a f^rogram'*^ 
functioning. Hie fjr-r. queriliun i mcern?^ rho /eliabililv of i.he 
:eaEur^s, sinn, it lik._lv tnac therc^ U some difrur^nro of 
o: Inion wicititi -ro^r-^n u tliu i a i rness^' ir ^'appropriaLun^^^" 
u n^ticula. pi durr . Su^ii iifferonces among respoadents 



up int€rna^ problums : .d result in little difference, or random 

ferances between programs. Because of this possibility, we will 

I 

be particularly inter^-^if-'H in rhP nsychomAtrlc characteristics' of 
these measures s especiE 'ly t. _i c r.iparison of response vari^ioii 
within and between programs* At the individual program level, it 
is prrticularly important to look at the distribution of item 
responses when Ijiterpreting results in this area, In ordgr to get 
the full flavor of respondent opinxon. 

' The sacOTid roscrviticr is mor^ HiffiruH to deal with, becausa 
it questions the appropriateness of participant juditnents li, fhis 
area. Are studenta tsveii ^dvan^ud doctcral student's, qualified to 
make Judgments .^ouc the quality of Llj£ir educational experience? 
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Can fjiwult.' men:bers riOnos'/ / : - : : ■ ; i : v : . ;r = '^ri^" :.:.ev 

helped shape? Obviot'^^ly there •lr^> no -ir-^l^.- an^=--"^r:^ :o suc:i qui>->tio'i- 
•Responses to opinion ^iurvf . d'.-- s-'i^^ii''-^- fL^flccL augerj fear^ 
JeFire to make :i pr gram i d. Al -q, c:.e validity ol responses 

to qucsrion? about currant "^oi;ra: ^roi.it^ n mav varv iDp.^nd3ng upon 
the perceived use of the in or- . : "^n , with perhaps les? open: ess or 
honesty If program survival is in luc.^tion". We would jnly point 
out chat J.ncerviews wi^h -students an" farulty members h.Hve the same 
weaknesses and* in . iLion, terui tc reflect the opinions'of Dnly 
a few of the Dartici] ints. A Iiigh response ^^ate on a well-designed 
sui ^ey insLrumtiPL j. i fers Lh- poLenLial oi feedbacK i r/r.. much 
larger and *more representative sample of part 1 cipants j and thet*fora 

as program functioning t evaluators should give close attention to 
response rates and should pursue inconsistent results for individual 
departments with more detailed progranr information or interviews. 
With appropriate csutlonss the approach should contribute to di^r 
overall understanding of quality in doctoral education. 

In summary i judgments abuut current program operation were 
Invastigated because it is difficult to belVe^ chat a program is 
academically excellent if students and faculty are seriously 
dig^%tlsfied with its content and procedures, even though other 
measures are high, ^n the other hand, high satisfaction with the 
academic program may raflcct modest expectations and could be quite 
cpnsis.. ent with lower measures in other areas. 



bv graduate =5i:!^'-'^i icr^^n- r .• ^' -.z ■ 'i--^:^ :..u:i;' 

ruri^icular offerin^^H, ^-va I u/: : i .mi oi ■ Mitzat orogre:^^ , d^r e rtquirt = 
ments, mo isfantshipa an.. oMiLrr work= rc 1 r ed ' =.dent experiences, 
relationships with connate pr^^^ram^ within the university, and j 3. 
piace^ ^nt jf graduates. In addicionj rru deans endorsed a measure 
of overall £ituaent ^atiHr^cLion witii the trducation they are receiving. 

faculty^ -i-- alutrini qucr:; t i onn :^ i rap ; respc'nd made Jud^nenr^ r^hnu^ 

each item on a four-point scale from "excellenL" to ''ptor. items 
concerning t'f same area were then grouped and ^he s.jores combined 
to f rm one scale score --^^sent the program level in that area. 

The six scales that demonstrated adequata reliability^ homogeniety, 
and discrimination betwaen programs (sea Appendix F) warei 

1 . Studant rating of thg curriculum , the program f laxibla 
enough to meet individual studant need^? Is tha curriculum of 
sufficlant breadth and dapth? Doe^ the structure or sequence 
of coursas follow logically? Is there sufficient opportunity 
for Independent study? Is thara opportunity to take courses 
outside the major field * and ara relationahipB good with other 
departmmts? (Studant Scale* No, 5) 

2, btudent rating of the relevance u£ de y rae requirements . Are 
rF-alrements clearly stated * relevant to admissions requirenients 



the field? 'ro qualifvinjc ^: i';.!:::.- a.:::, i . t e r - lairly; -r-. 
language raqiiirGment?^ and r-quirt- ; . .^.ut^^^^ i :'":ttr ri-t' 
appt opr ia Lij- ' (Stude^nt : "al^; . 

J * C a 1 1 1 y _c t u d cnc exp c r i en c e s a a r e s e ar c ri o r te ach iri^ 
assistant , Do assistantsh ips contributG to student acadeinic 
d eve lopTnen t ^ Do t f?v :i; i e s t ud en 1 a chan j acf in a 
profeHsi on a 1 r o 1 i p t wo rV relevant to antictpat ed later 
professional dutii* ;? Are ass ants treated as collii.apues by 
mender... of thp d larLmen r'^e ass istantships fairly administered, 
and are there enough their ■ adequate pay? Is there good 
supervision? Is there ^-ufflcienc office space and equipment? 
cudent acaie Nu* 

4, Stud^aL uvHrall sa tisf ac Lluu with Lh« prugr^m. Do btuutsntte fe^l 
they are getting good preparation : or their later professional 
work? Do they think they have learned a great dr-^l in the 
program? Wou^'d they advise a friend with similar intarests 

CO enroii in cne pragram? wouia cnay transiar if tney couia 
do so without losing much time to the degr^^-^*^ (Student Scale 
No. 3) 

5 . Fa cu 1 1 y j u d pie n t s abo ut t he^_ade£u i c y of student evalu ation 
prn cadure s ^ Are degree requirements administered fairly? Does 

j 

the departinent have good procedures for evaluatiu^ tudent 

ogress toward the dt^^ree, evaluating comprehensive exam 

performance, and supecvising dissertations? (Faculty Scale No. 6) 

•» 

# ■ - 
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6, ^tfflol satisfaction with dlseertatlon experiences . Did the 
^ ■ student have sufficient freedom to select his or her own topic? 
- Was the es^ected ycopc of the research pro^^^ satisfactory? 

piuyfedui^^ ruj; ^elecUing committee members and arrange- 
menl for interaction with the cOBimlttee satisfactory?* Was 
there sufficient opportunity for the atudejit's creative thinking 
and individual eKpresslon? Was the dissertation ^xperience 
related to ottiec. prof esslonal skills and to career plans? 
(Alumni Scale NOi 1) * , ^ 

Table 9.1 presents correlations among the six Scales by disci- 
pline, and klso the a^^erafe department mean score and the standard 
deviacion of variation ij program scpres for 'each scale. The means 
and standard deviations iof the first three ^calaa wggest that the 
HLuutiiL^ In Mw^t programs in all three di^clpllnel rate program 
contents and proce lures "fair" to "good''"a rang# of roughly 2.2 
to 3.1 when a "fair" rating is scored 2 .and a "good" rating Is 
scored 3* -Though the differences between progrOTS are not large, 
the statistical analyses of the scales Irtdlcate . that they are 
stable and significant. In general, students rate thair personal 
satisfaction with the program "good" (table Item no* 4) and alMmif 
average a high level of satisfaction with their dissertation expei L- 
ences (Item no. 6, where "satisfactory as is" ,wa8 scored 3 and ■ 
"could Improved" was scored 2). The scales appear to operate 
<^ery similarly iti each of the three disciplines, even thougli the 
content of the currlculima arid exact progrOT procedures are quite, 
different* 

^04 
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The aorrelstiona* in Table 9.1 Indicate that all the scales 
are positively related to one another,, with a r articular ly strong 
relatloiishlp raong student Judgments about various aspects of the 
progrlsm Operation. Program flexibility,^ course rln^i and 

. . ^ ; ^ ' _ . \ 

relevance of degree rsquirements appear to 'be especially impoitant 
to student overall satisfaction with the progrMi In history and 
psychology, wh^ireag asslstanshlp experiences are .more jjoportant-to 
Student overall sat^^attlon It^ cHemietry. Faculty satisfaction 
with the departmental practices^^or^ evaluating students are pogl- 
tlvely but much Ipss strongly related to student judgments about 
the program, particularly In history* Alunml Judgments abjut* 
dissertation experiences a^^.not related in any consistent way 
with the other ratltigSi On baJance, the six scales seem to "hang 
together" *aB measures of an Important area gf program' excellence i 
throt^h lo*^^ relationships between student measures and measures 
from the different perspectives of faculty and almml raise some 
questions that n^eed further, exploration * 

Correlates of^ Program Ratings ^ 

-do measures of current program functioning relate to other 

'in 

^dlcatms of progrMa quality? Table 9,2 lQ0.m at the relationship 
betw^m three student ratings of the program (currlculLai^ degree 
requlrments I and overall satisf actlori) and other program descriptors 
selected because they seem logically related to program operation or 
because earlier chapters Identified theja as particularly good 
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Ind^ ators nf prf*^raffl quality in other araas* Intei^i.i^ing 'these 
correlations i It Is apparent that students who th\nk that the teaching 
la very good and that the students and faculty are very capable^-^tema 0 
and 3) also think highly of the curriculum and decree requirements ^ 
and in general are satisfied with the training they are receiving. 
Student Judgments about the program operation tend to have a low 
pOiitlve or negligible relatio^hip with KOit of the other variables^ 
on the list-^-^peer ratinf of the faculty ^ siie, level of emphasis on 
teaching} percent of faculty with outside research grants , average 
faculty days away from campus* This suggests that these characteristics 
have little bearing on what students* think of their programnj 
particularly in chemistry and history. Correlation of student Jud^ents 
with the' adsquacy of the physical and financial resources and relative 
level of faculty professional experience table item #9 and 10) 
suggest that these characteristics are positively related to student 
views about thair programs in chemistry and psychology, but neutral 
or negatively related to student views in history. 

Student ratings of assistantships^ alimnl satisfaction with 
their dissertation experiences, and faculty Judgments about departmental 
pr^.ctlces for the evaluation of students present somewhat more 
complex relationships. The way they are correlated with a variety 
of program descriptors is reported in Table 9,3 , In this table ^ 
we see that student views about asslstantships in chfjnlitry and 
psychology are related to the acadralc excellence of the progranip 
Its slEeg and its emphasis on res ear ch; but, in history^ 
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*Stafootnoti». Table 9,2 ' f ' 

Notg; Diicrlptors were silictid for thair rtlivince to thioe particular itudent ratingi; thsrsfor 
some of than art dlffarint from the discrlptors included In Table 9.2, 



thte pii^LuiiM iei quiLe uirr^ittULv views o£ history assiscantshlps 
are positively related to chardcterlstl^^ of the environment am' 
aegatively related to research emphasis c reputation. Some of 
this dlfterence may result from the combination of raaotldn? to 
r^fteareb and tegrtlng acsistantsh Lps in tht same set cf itsma (aaa- 
pag« 3 of the .student questionn aire in Appendix D), The departments of 
chemistry and psychology probably Dff*r fair nmsfcers of both types 
of asslstantshlps, so that they are welghtad fairly equally in the ^ 
final department index. But in history, almost all of the atdlstant- 
ships are in teaching. Alsos more students in histor^ are *preparinj* 
to become teachers* Therefore, the correlr ^.ions could be Interpreted 
to mean that views about teaching asslstmtihlps wong students 
preparing fot a career as a teacher are essentially unrelated to 
the publications racord and rg;^utatlon of the faculty but do refleat 
the clijoate and human relatio.iships of the department, whereas the 
opposite ^s true for views ^rhout assistantahlps in the more research-- 
oriented fields of chemistry and psychology* ^aln, we see that 
teaching and research reinforce one mother In the laboratory 
sciences (chemistry and psychology) but not Jji the htmanltles 
(history). ^ 

Alumi satisfaction with dissertation eKperlenees do not appear 
to be very closely related to any of* the other program descrlptbra 
listed in Table 9.3* with the possible ixceptlon of overall ratings 
of ocad^iic excellence (items 3 and 4). Theic low correlations 
prob^ly reflect the generally high ratings given by alunnl to 
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their dlaserta^lon expariances, and the sieall variation mong 
pragrina on thl^ seore (see' it^ #6 In Table 9.1),s AisQ» thi* 
^ rating reliiAility is not aa high for this scale as for the others i 
and the sigAlfieance of tha difference beween programs is ^ 
quite so certain (see Appendix f ) , Frobubly rh^ icems in chis 
scale would be more UAefal in feedback repgrto to individual 
prograp^ than' conbined Into a scale -see re for comparison across 
^rograasi . 

# 

Vie%fB Abdut Job rlacOTent 

toother area of program functioning that was endorsed as 
Important by the graduate deans is job^ placement of new graduates* 
^ ^ Doctoral faculty m#.mberi take prid# in placing their top graduates 
with prestigious departmants at other universities, and in general 
helping oraduates through th^ "old boy" network of person .1 *acquain** ^ 
tances. The decline in academic Jobs makan this syst^ increasingly 
inadequate p however, and many departments are taking more formal 
steps to legate a broader variety of job opportunltliBS« Also^ more 
departments arr encouraging students to consider nonacbdeoic careers • 
Because the placenient of new graduates 1b 1ti fluXj it ii no*! a 
reliable indicator of program quality at the present time. It 
continues tV-^^^an area of Interest, however p and* responses to 
qu^tlons, in this study concerning placement vere tabulated for 
descriptive Information about programs. A summary of some of these 
results is reported In this section. 
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Mmv students expressed concern about jnb pla(.9raent In comjaente 
that %imt^ volunteered at the end of their questionnaires. They 
voiced particular seiiSL tlvlty to thi» role of progrm ^ d university 
prestige in relation to jab placement; coranents such as "the faculty 
here do tiot have thf* nHtlonal reputatlor*a or contacts neea^e *3 
help graduates find good Jobs" or "my orly regret Is thac It will 
be even harder to find a Job when 1 graduate because the department 
Ig flot\»ell known" were conffion. The pervasiveness wf concern about 
emplo^raenc. probably contributes to the willingness of students to 
put up wlch uncomfortable envi^wi^e^ In exchange for program ^ 
prestige, as observed in earlier discussions. Differential percep- 
tion of placement helpfulness from the departments with different 
reputations is also reflected In the questlonnalra responses that 
are reported In Table 9*4. An average of more than 40% of the 
faculty In high-prestige departTfientR think the department helpfulness 
in jobr placement Is "excellent^' (item 1), buc only an average of 

about' 15% of the faculty in low-rated departments feel the sms way* 

I 

Apparently all departments make some effort in this direction, 
however, since about 40% pf the faculty across all reputatlonal level 
say the department -a helpfulness is "good." 

Students are, somewhat less sanguine about the placement help 
they wtll receive from the department ar the faculty (It^s 2 and 3, 
in Table 9.4) even at top- prestige pLOgrams. Roughly twd-thirds of 
the faculty in each discipline think that the depirtmental plaeement 
efforts are at least "good," but only half of the students fifave 
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Vifwi About Job Placetnent lAverages oi rtograi i^eicentapBi 



1.. Ficuityi Bipirtmiiitii lielpfylnesB la 
job ^laeniBt 
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, piiclni paduatea 
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PsyehelfgK 
M L 

28 11 
44 40 

28 11. 5 
39 42 ,31 
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FsyehgloM 
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See footnoti, Tabli 5.2. 
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this Impression. In genermli perception of helpfulness decllnee 
with vapt^mtlonal rating, except that low-rated programs in hj story 




it 



praatlcas' mnd experiences in relet cSffilnt* ThelW ■ 

responses are reported In item 5 of Table 9.4, Since tWese tabula-- 
tlons refleab^nly one response per program, they have not been 
separated into reputational levels* The chairmen of history depart- 
ments, particularly 5^ reflect a high level of concern about job 
j>lacaient for their graduates. More than 70% of these departments 
raaintain placement files in addition to encouraging registration 
wi.th the imlveralfy placement office, have a departmental coimittee 
^ coneamed with Job placement, and consider placemenf help to be oiie 
of the Important functions of a dissertation chairman. Every 
history department In the study reported that placemen u of graduates 
in appropriate academic positions has b^ome more difficult in the 
last two years. The iepartTOents of chemistry nnd psychology are 
not far behind* In addition ^ twn-f .irds of the chemlstrv chairtnen 
* raported that placement in appropriate nonacademlc positions has 
aisQ^^^ome more difficult In the past two years* 

Obviously successful job placement of new graduates is one 
indication of an affective graduate progrM, though In a very tight 
job narkat short-run indicators for this dimension could be very 
' mlslaading. Elacements cannot be made In positions that do not,exlRt; 
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th« reputation of a program may play an es^aclally important role 

ffrst job offers # For these and gther reasons » probably some 
avaluatlon of the ©ffbrts that are being made to broaden options 
md prepars students for a greater variety of etoployment possibilities 
are as Iffiportmt as an aetual tabulation of ^ttl^lnda of jobs that 
graduates are aecepting whan thll area of activity Is investigated as 
part of a progrra evaluation. 

Uses of Aeadmie Progrm Measures 

Host of the program measures revised in this chapter have been 
scale sc4^^ to represent characterlstles of a program'sl academic 
offerings and prucedures* The research r^qults ^Indicate that many of 
these scores are related to measures of other progrra characterlsclcs, 
especially if the assessments ate made by the swae observers. Tha 
research Laa det^mined that reliable measures are available in this 
areai but we^have not yet addressed the question of how they might 
be used. 

■ 

Ratings of progrM functioning add the important dimension of 
process to the assessment of quality. At the level of individual 
progrm review p a great deal of information can be obtained about a 
program's structure and contents. Unfortunately, such anecdotal 

material does not lend itself to comparisons across progr^os in, 

/ - - - _ 

different disciplines. The "adequacy" measures described in thli 



chapter suggest one way to circumvent this problCT* They provide 
useful comparisons of Judgments about program functioning, although 
questions of Internal program equit^^ill be viewed by many as 
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father tangential to questions of program quality. Fgr this reason, 
the separate items In each Scale (enumercited in Appendlif F) may be 
more useful as feedback information to Individual programs than as 
comparative progrra assessmentSp indicating particular strengths and 
weaknesses as they mre perceived by students and faculty meters. 
For eximplei opinions of progrm participants about the fl^lbility 
of the curriculiHn to meet individual needs g oppprtunlty to take 
courses In other departments i clarity of degree requirenents, and 
fairness of administering assistantihips :an. be very useful information 
for a program that is looking for ways to Increase its strength^ 
without any need to reduce the data by forming scales. ^ 

Figure 9,1 graphically ^presents responses to four individual 
items from scales in this area with res as grouped aacordlng to 
high, mediums and low peer ratings of faculty quality within each 
discipline. The itcmc selected fsr illustration are ones that . 
correlate particularly highly' with their scale scores • The figure 
demonstrates the kind of discipline or unlversity'-level comparative 
data that could be provided to help departments interpret their 
own responses without making any value Judgments about desirable 
levels or distribution of responses for a particular progrw. 

In terms of overall level in Figure 9.1j students ?n history 
programs give the hlghebt percentage of "excellent^ or "good" ^ 
raftings to curricular flexibility , relevance of degree requlrttentSi 
and opportunity for assistantship holders to act in a profesaional 
role. Chemists are least satisfied with these Aspects of their 

Sly ^ 
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Figure 9.1, Selected individuar ite:- R^:^;^:.ih^ 
the Academic Progr^- and Procedu^ 



Student rating of the 
eurrleulumi Flexibility 
of pfogrm raquirMenti 
to MSt Individual 



Student rating 
degres requirements* 

Relevance 

ol requlrCTenta to 

t t ^3 f. ft 4 - J 

the field. (S-»4-l) 

Q} 



Student rating of 
aisistmtshlp ex-- 
pericncss ' 

Chmee for (teaching 

or research) assls= 

tant to act In a 

professional rwla. ^ 

(S^S^e) S 

u 



Faculty rating of 
evaluation practices: 
Depntment 
proeedures for 
evaluating graduate 
student progress* ^ 
(F-2-0) 

d) 
9^ 



H i Hf.er . 



lb 
70 

la 
/a 

a - 



r 

7© 

JO 

o 



TO 

fa 

¥9 

•J0\ 



n 
1 



\^ 

I ^ !l 3 li 

3 w £ w ^ w 



H M L 



n 



IS 




^ 

Peer Rating 



1^ 



H M L 
Peer Rating 



n 



iii 



II M T 



Peer Rating 



H M L 



H M L 



r 



programs* On the other hand, his Lory faculcy arc least satisfies 
wirh departtaental procedurfeb :wr aval acing graduate student progress. 

In geneial, the oar f^aph^. support our impression from other 
analyses that Individual ' t^ms , as w '1 sc.ile ^coLes concerned 
with prcgram structure and :^rocedures. are appropriate f ir use in a 
number of different disciplines and across all lavel&'of program 
size and reputation. i he . Tii o i^y of responses tr eacn of the items 
in all three disc 11. .cs sugr sts :;iat 11 would not be necessary to 
collect separate comparative data lor each discipline before u^ing 
inaiviauax item:: zz: ccr:cris; prcgrarra in tnic area* A^j:., tn- 
similarity of responses across reputational groups IndlcateF that 
score differences between programs reflect different expectations 
and tht: respOiiuerit ' s pert^eptiwus uf pi Lli.;^^ and pi- .w^at^^ raLher 
than haio or other reputational bias. 
f 

4- 
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ALU>D^I ACCOM^^ ' .iiMENTS AND OPINIONS 

Most of the dai discus:^td so far in t report came fron, 
current program partic^pantn---.idvan^cd graduate students and faculty 

~ ~ . ^ - . . - ..... 1 . ^ - - - 

These sources of i- forrn^qtinn w»=^re -q position to "re^l it l.tVe it 
is" in each department in tru- spring ol 19 75: No time lag tor 
changing conditions to catch ui with reputat^' ^n, no unidentified 
halo effect, no second-hand guessing bout what goes on behind t; 
facade. These sources of inionnat^-jn have been emphasized because 
they are readily available as well as appropriate ^'or program 
assessment. There i& one pther source of information about programs 
that is highly regarded. If seldon surveyed, and that i their alumni 
Many aepartment chairmen and other scholars of higher aducacion 
will, when pressed to identify an ultimate criterion of program 
quality, state that accomplishment of the program-s graduates is 
the best single measure of its success (cf.* Willingham, 1974)* 
The graduates are a tangib^le product of the program; they reflect 
the abxlities and skills they already possessed at entrance^ to be 
sure, but they also reflect the direct and indire influences of 
their experiences Li graduate studtmts in a particular degree 
program. Chairmp= and faculty mc.becs of programs, therefore, feel 
LUey iittve a legiLiiiiaLe txgnu to cxaiiu te*uiii« cieulL lul the kiiiu.^ ui 
jobs their graduates hold, the scholarly works they produce , and 
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the recognition tr.ev receive in Lheir careers. A....: ':-\iev^ 
that it would 3 appropriatfc to judge the quaiicy of the prograin 
by these products. 

in addition, it is or ten suggestad tnat recent aiusni wiii 
havfe more perspective cibuut cht: couLt^nuB, pijced^re^s did .^:uLiii- 
r^p^i^^ of n nrnr^n^ f-hn^ r^ = ^rr^^*^^^^ pnr^llpH £7 1 i_id ^= s : -r^n^j^ 

objectivitv than . facul tv menber^ ; ant' -ore information on whicli 
*o base, iudgniunt-^ -ban h. colleagues in the fieh' ^ .0 have ncve^ 
studied or worked ii. . e departnit^nt . Thvr^(u.:e, theii views about 
their graduate prograr. are also valued is feedback tor program 
iiT'p^vernenl ^ 

For both of these -isons, inforir ti^ a abou. each program 
from a sampl^"^ of recent alumni w^s considered an essenL.al part 
of this studv. wanted to hear from alumni who had been in jobs 

^ * ^ , * . . 1 r , , ^ _ „ T j t ^ *= V , , ^ 1 ^ ^ ^ u ^ u - j 

graduated recently enough to know something first-hand about the 
program* A: cer discussion witin exparcs in each discipl4.iie , 11 
was decided to concentrate cn graduates three to five years after 
the degree (those who received the doctorate in iy/U, 1971, or 1972). 
Participating departments were asked to provide names and addressee' 
of persons who received doctoral degrees In these years. Almost 
all of them Hid sn. Dues t i nnna res and one postcard follow-up 
were mailed to all of the continental United States addresses that/" 
i^^r^ qv^n^hl^. Mjg^v rin-*^f=lnnnai r^R were ret!irned stampetf unda^ / 
liverable; it is knowii that others did not reach the right person » 



detpite the ""act tli&t they 



- H ^r, - J=i --^ 



received for l,!/^ aiumni, or -tn o :;u.;.5t: wm^^^e cnv e xupt?-^ vtcit; 

returned to us , ^ 

Th is c h d p t e r ^ j v m n ;-i r t i , i ^ l r L t # n c ^ o Pi r,. o u s 1 1 o ns a t 
adequacy of this samp to represent alunini accoTnplishments and views 
about programs becHUse the re. "Hinse rate v^as not as hi^h as desired. 
However J ^. t probably reprGsents a realistic return, the kind that 
^could be expected if de^ar Ln..intH u-ere to survey their own ^^.raduaces 
with a sirnilar questionnaire. In fict, otee of thtse qiiestionnaires 

their average rate of return was about ^he same* It is aleo likely 
that the reactions of some alumni are i.^cludcd in the study as 
faculty Tather: li u eiluiuui i espouses, because th^y arc currently 
employed in the department that granted their highest dagree. 

Alumni Rasponses 

Usable questionnaires from pre ;ram alumni were examined to 

aatimata response bias and to evaluate their generaliEabllity 

before investigating their relationship to other measures of 



^Departmeaf^.s were adked to supply the names and addresses of all 
( ocCoral degree recipients in 19/U, iy/1, and 1972, A faw programs 
i ould supply the information only for one or two of these ye* Most 
ild not have addresses for at least a few of the graduates in this time 
period. In the few cabes where more than 100 degrees were "varded In 
this period, questionnaires wer^. sent to a random sample of 100 degree 
recipients. Graduates of two programs in each field wera not surveyed 
because fewer than 10 degr- es were awarded in 1970^1972, Thus, 
cecnnicajLiy Liie autv^y v^^i^ llu^ tti^ ^i.^^^^^^^ -^^^ ^^^^^^^^^ ^ 
programs three to five years after receiving tha degree, but 'it came 
close. 



program, characta^is tics . Firytj at tie ::_;nirer of alurni 

recurris froin eacn prograr: in relation to ne n-.iriber uf quus c iunna ires 
sant out. AiL..niii responses ra used only if there vere at ''east 
i 0 us ab 1 e r *^ p 1 i e ^ h n d f f - \, ; . , ^ m n f «if <] rno r h l I * a n 2 5 % o the total 
numbar mailed out. or if fhere were fewer than 10 usable replies 
but they represented more than half nf all the questionnaires mailed 
out. The latter ^^-i terion wa- appropriate for a few programs that 
granted a small nuiubei of degrees between 19/ v. and 1972* Alumni 
returns met one of tb° e criteria for 20 chemistry prograiris, IB 

and confflaents aboiit alumni responses are limited to this sjLightly 

smaller set of programs* 

Am menLiCMied eailietj Lh^ uv^£^ll iluiiuil r^spuiise rate was 

about 48%, The average response rates for individual programs 
c 

ramainlng in analysis ^ grouped accoraing to peer ratings of 

2 

faculty quality, are presented on the first line of Table 10. !• 
T..e^e are verv nDnRervative rat. s of return^ since thev co Id not 
be corrected for undeliverable mail. Response rates were slightly 
higher for lower-rated prngram^^ bur essentially Quite similar 
^across disciplines and reputations. A slightly higher response 



2 

The smaller number of programs with aluml responses available 
for analysis necessitated some regrouping of programs In the ''madlim' 
and "low" reputatlonal levels tised In this chapter. For al^ml, "low" 
means m riputatlonal rating below 2,4 rather than 2.0. This results 
- la fewer programs in the "mrllum" category and a fairly even dlstrl- 
b^lon of programs across the reputatlonal levels i 6^8-6 In chemistry j 

In history; and 6-6-^ in psyrhnlogy. Se# nhapter 2 for a 
dlficussian of the rationale for grouping progrms In this way. 
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Groupings for aluiml d^ta are as folic .^s* See footnote* Table for 
groupings for student data* 

H ^ High peer rating of faculty quality (above 3*6), N-6 programs in 
chemistry and psychology, 5 in history. 

M ^ Mediina p^er rating of faculty quality (3*6 to 2*4). N=8 prograins in 
chemistry and history , 6 in psychnln?|y, 

chemistry, 5 in history and psychology. 



rate in .nver = rateJ pro^ra-.-- was -':ic:or.&d, -ince these Alt^ ■ 

tended to Dt : ne sn^li^- L^wgLai.ij? -iiii i^^ci^ aJui^r.i 
CO .represent each pr .gram. In terms of the actual number 
nf responses -ed in rl, alvs^-,. 5 respondents oui of 9 alunini 
rapres^mt 1 -ro^r i.; onr extre-' and 68 rp ^po^^ ^ien t s ou^ '"^^ 100 
aluTimi at the o her. MosC program: mean scores represent tb 
respnnses of rvetween in and 35 altinni. 

We do not know Liie char.i .^^ristii.s ..f .11 197^--ly72 doctoral 
degree recipientis of programs In Lriis study ^ luC questionnaires 
teLyi^iiKd l.j -w..t ^...:^ran^iy jnrr;:.ic^ anvan^cc sraauii-- bi'---n-- 
provide a profile of these '^non-ta=be aluinni* One check on the 
representativeness of alumni resp'-nden ts , therefore, is to c^ Tapare 
che characterisLLcs uT Lhef.e Lwo ^roup^. We would not expect thein 
to be identical I since the enrollment pattens of some of the 
programs may have changed ove^', the past six or eight years ^ but 
we would expect alumni to more like current students in fhe same 
prrigram fhan 1 i sttident s I n Other proeTams . Table 10,1 also 
reports the results rf this analysis. 

items 3i and 4 in rnhle 10,1 compare alumni and enrolled 
Student respondents on sex, race, and undergraduate grade-point 
avarage. Most of the alumni respondents entered these programs in 
thfe middle 1960s; most )f the student respondents started their 
graduate studies in the early 1970s, Therifore, an average of about 
7% more %9^mn and 8% fewer Caucasians in the current student gopula^ 
tion probably represents a reascnable shift in progrMi enrollment 



patterns during this p. ' a. AL;^^., ■ . it-w i n/i'ion w.s '.'uie 

included in tb ^lin;:'! ' 11^ , vhic^ ■ :^uid tend t reduce - 
ethnic divarsity. Slightlv hi ::h i proporfi)ns of cutrent s^uaentr 
report undergraduate ^t^:: ^ a- pc^nt n-'ar .;gs 3=:^ cr ibDVei whic i is 
conslscent with rhe r.^end t^-^^^-r ' u-ir'^ -graduate "grade infl^^^ion." 
In general, however, the verze are similar and cons:.stent be*:w^en 
al'imni and students '^n nil t r o variables at nach level wf program 
prestige and across nil thro,- iiHuinllnes. 

Programs ilso varv In • h^- ill^iti ns the^-' ufferj ard the 

proportion of stuoents W'j feipeciax^^e in ^ givfeu area. Therefore, 
the final comparison betwee i alumni and student respondents is 
concarnad with the percent who identify themselves with two popular 
spacializations within each discipline. The percent of alunmi 
raapondents are compared with the perc nt of currently enrollad 
raspoTidentR ^n nrffanic and nnvslcal chemistrv, modern and American 
history, and axperimental and clinical psychology* Though not ^ 
identical, again the patte'^ns ar^ ^-I'^n^r at^A i-nnei^tRnf ^rrn^R 
groupings * 

From this evaluation of the alunini response /atcs and a comparison 
of the respondents with advanced graduate %tudent re^DOndents fr^^m 
the same programs , we concluded that the alumni data were sufficiently 
representative to be useful as program descriptors and in analyses 
with other variables. They may fail t give a coTTiplete picture of 
the products of each of the 55 program^ that provided usable alumni 
data, but they represent a unique resource in providing comparable 



inforMtlnn ab..at a num- er alur^nl ivorr. se'/eral doctoral programs' 

©Ech ot the three d i s ^2 1 p 1 1 hq-S , '1 , 1 t iv 1 1 ^ es and o in iun^ ^.^ J 
thase alumni are discussed in th?. ^e:r .inder of triis ch^ jter/ 

^ploymfcat of Alumni 

An tarlier chapter aiscussed the purposes of doccoi ^ ^'^''gtams 

i i = er IT.? of the ^.T 2. t ' -^^n ^ r H C n " r-, ^ ^ ^ r- n ^ ^- K .-^ " r- ^ ^ ^" 

teachers, or othir professional practitioners, and concluded .^lat 
the faculties of mos - programs perceive their mission 10 be the 
preparation of scholars and/or researchers. Thereforej one would 
axpect most of the gradu^t^^s of these programs to be employed in 
uiiiversiLy pubiLiuns ur as tesearchers in oqher setting?. Table 10,2 
preaenta the average percent of alumni from programa at three different 
levels of prestige fpeer ratings of faculty quality) who reported 
first position after the degree, current position, primary work 
Mccivj-tiy m the preeent positions an two judfe=aents about their 
present jobs* Conclusions that can be drawn^from these data about 
the employinent of recent alumni In three fleias are that ^ 

***flfeout half of the new chemistry graduates of all progrMs had 
poscdoccoral fallowships Imiediately rfter receiving the degree* 
Host of them appeared to have moved Into research positions or 
Into college or university teaching by 3 to 5 years later. 
An i^Fu^Hgp of 60 to 70% of the graduates of all chemistry 
programs reported that they were currently engaged in research 
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footnote, Table 10.1. 



Nocf! ^Percentages lay not add to 100 because adjunct faculty poiltlons ind eiployient in id-inls. 
iV£ posliions, cleffieiiUijf ui bccuiidary achoois, oi "ottiir an not tabulitid. fmt thm 41 of the 



alujsni froi any program reported that currintly thiy weri not eiployid for pay, 
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, the largest pe^ jent nf ^ruduate? fro-: hisi jr;- prograud took 
first jobs in four^yaar csliegaa, and the inrgest perce;., 
continue to b*^ employed in this setting, Only top^prestigQ 
history programs plr= od a' many as one-^thir-'' of c'^^-ir graduates 
in Ph.D.=gr uing universities^. More than'hal. of the history 
graduates in a^i programs reported veaching as their current 
primary job activity. 

graduates of ps--cholog- depar tme-nt s spread out to a greater 
vHriety of positions than did graduates in the other two fields, 
both immediately after the degree and later* Roughly on?- third 
were employed ny colleges or univarslLles and one^third in 
research positions or professional practice (typB of employer 
was not related to program prestige). However, about half of 
the graduates of highly rated psychology prograins reported 
research or researcn and teaching as Lhe piliu^L> u^Ai^ut Job 
activity! while the percent in research was smewhat lower 
for lower^rated programs. 

.fewer cheniists thsn historian? or p??ycbnl ogi sts reported that 
their present job was "highly related'* to their graduate fielv 
of study. 

,an average of about 10 tj 25% of these respondents who recelvad 
degrees from doctoral programs in 1970 to 19; ' '^ported that 
they considered th^'^^^''^^^^ nnd^rpmployed becane . f:helr present 
positions were not in their field or not c*>p. distent with their 



level of training and t^xpariGnce, T^.g f :^^^c&ntages are somewhat 
lar-^er in low-rated chemistry deparcrit^n ls and in high^ or 
mediuni-rated history and psycholugy depf^rtments • 

In Bleary, the only clear differentiation by prestige of tha 
degree-granting programs that appears in Table 10.2 is the higher 
Ht^ictf-L uf gr^auares empioyea 1 / Ph . D. -granting universxtieb , and a 
corruspQn.ingly hightfr p^i,,. r of graduaLes from the^e pr:3grams who 
report resaa. jh or research am: tea hing their primarv current 

praatlgious programs disperse much more broadly than they did a 3 ww y^ari 
ago, it Is still true that most permanent positions at major univert dies 
ara fillad by graduates of what are considered to he the top programs/ 
in tha field, ' Ivmni views v^out the relevance and approprlatenasay 
of their current employmeni re n t related to my appreciable ^ 
extant to tha prestige of the program from which they raceived^-^eir 
dagreas, ^ 

Expected differences in the employment patterns of graduates 
from each of the disciplines were fotmd. Chem^^?:^ v^re most likely 
to be employed as researchers, r ^ychologi sts as college 01 university 
faculty or in other practicr, and historians as college 

teachevj * 

Alumni also were asked to rate a number of statements concerning 
their employment setting, climate, and job satisfaction. These 
resul 3 are praaentad by discipline and peer rating of the graduata 
faculty in Table 10.3. The items reflect: an assumption that most 
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Tab la 10. 3 



Alunffii DtsMlptlon of Current PoaltlOT (Avarsge of Program 
' Psresntagts Responding "Very Descriptive") 



Chsmisfry History Psychology 

H M L* H H H H L* 



i; It provides mAy oppor- 

timities for research and « ,a y i i/ 

creative work CA-17-b) 48 38 34 29 25 18 41 34 35 

2, It does not use all my 

education and skills i/ io 

(A-17-c) 13 17 19 19 22 16 14 23 19 

3i It Allows mm to Increase 

my visibility within the /a 
pLfeaslon (Ll7-f) 43 33 40 27 23 17 40 30 39 

4. ^ colleagues are first- 
rate sfcholars or re- ' ,^ na 

, searchers (A-17-h) 32 26 26 11 13 21 26 15 19 

5. .It gives nJe a feeling of 

worthwhile accompllstoer- t, 

(A-17-*)' 61 ^9 52 45 49 51 52 55 63 

6. It is a very secure job 

' (A-17-d) 27 28 27 28 29 42 27 30 42 

7. It provides a very cam-w *^ l,« ^/ 
fortable salary (A-17-a) 35 35 30 22 16 ko 31 34 34 

8. It gives me a chance to 

exercise Isadarshlp ci 
(A-17-J) *3 35 42 28 33 35 51 43 58 



*See footnote. Table 10.1. 
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f Acuity maib€]f8 111 the disGipllnes surveyed in this study would agree 
that a good Job fui Qim wl thi^ii pruLegef would offer an opportunity 
tot ertatlve work and research, utilize the peraon's abilities 
provlda eondltlona for advancement in the profession, involve contact 
with other first-rate scholars, and result In a feeling of worthwhile 
accosplishnant. Tha first five itms In Trtle 10,4 give the average 
percent of re^ondents in each of the prdg^am groupings who said 
^that each of these statements was "very descriptive" of thelx current 
work« The higher proportion of chemists and psychologists who are 
aployed as researchers piubably accounts for the sonewhat higher 
level of their responses to the first que^stlon (coticsrnlng. opportunity 
for research and creative work) ^ and for some association betVeen 
response on this question and the reputatlonal grouping of the 
departeent. Responses to the other four questions are uniform across 
reputatlonal levels « 

The last three questions in Table 10.4 alsb reflect important 
Job considerations for a young professional, but they are no^t 
necassarlly related to the scholarly tradltioni job security, 
salary, and opportunities for leadership. Responses to these 
questions also are uniform across reputational levels, 'with the 
possible exception of the salaries of historians, 

Alifflini Caree^ Accomplishments 

Since, as noted at the beginning of this chapter, career 

accomplishments of alimn^ have high acceptance as a criterion of 

■ 

* 
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doetotal prdgram quality* alwmi were asked to report their profes-^ 
sional activities to t number of areas. Riay also avaluated the 
excellence of the career training they received as a part of their 
graduate studies, and made Judgments about the overall ^xcellenee 
of their graduate progrms. Responses of the aluBmi from each 
program were suraftarlzed^ and the mean score gr percent pf aluaml 
giving a particular response to each item was adopted as ,the program 
IndicatoP on'^that variable, ^ 

Table 10.4 presents the major questionnaire Items concerned 
with aluTOii accorapliahments and program outcomes together with the 
means of departmental mean senras or percents* their standard 
deviation p and cutfelatlons among the variables* Discipline 
variations in employment patterns, reported earlier. , are also 

ft 

apparent here; in addition * there are disc;, line differences in 
alumi satisfaction with their career preparation (#1) » the percent 
who reported post degree research on their dissertation topic (^4), 
the percent who had published part of the dissertation (#5), the 
percent who had published a book (#8), average nirabar of profes- 
sional presentations (W), and current annual iacome (#6). Earlier 
chapteri In this report inalcate that many input and process char- 
acteristics of doctoral programs operate quite similarly In rather 
diverse fields of study; these data from al^ii suggeit that the 
(valuation of program outcomes In terns of the ectivltlas and 
* . accomplishments of graduates need to be viewed separately by 

discipline or area of study, 

o 
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Table 10,4 ^ 

IsCeresri-eiaCloni of Alumi Caracr Ac ^^pliihmencs and Jud^ents 



Cor rt la £ lofts Hman ot 
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8 
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69 
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a. 


fsf cent vlth €ut^^t a^dMle 


C 


38 


— 


















11 


.10 




^pgliitinfc at FhtDi tmlvartJty 


H 


33 




















25 


.18 






p 






















23 


• 14 


3, 


fmscMt with fMcareh et rase arch 


c 


u5 


21 


— 
















63 
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saA CMGhlmg aa ptiaary ver^^ 


H 


-10 


72 


















30 


,17 




aetlvltir (A^l8) 


p 




1 i. 


















41 


• 17 


4* 


farecnt 1^ raporfcad furchar 


C 


-11 


19 


26 


— 














37 


.10 




raaaarch m diaaartatlon 


H . 


. -10 


21 


-05 










• 






78 


• 17 






p 

£ 






IL 
1 ** 
















Aft 

as 


■ ID 




Parcant utio had pybllahad pare of 


c 


16 


16 


14 


10 














82 


,13 




diaaaftaciBB (A^23} 


H 


06 


30 


11 


67 


— 












Sf 


*15 






r 


£n 

pU 


pi 


1 ^ 


















*le 


6. 


Avaraga antival Xn^rmm tfi 


C 


-14 




20 


-22 


06 












17. P 


1*8 




thouaantt <A^26) 


H 


-17 


-04 


-31 


73 


41 


— 










15*1 


1*5 






p 


~m 


"IS 


"=*ip 


— * 


— jj 












19. 1 


1*7 


7, 


Avai^aga nuBbat ef Journal 


C 


25 


33 


26 


46 


-11 


-01 


— 








6*54 


2.03 




publlcatifms (A^27^at b) 


H 


-11 


20 


29 


-02 


-27 


-21 










6.37 


4*12 






r 


J / 


t\j 


/o 


AO 


Aa 












3*4^ 


£ . 10 


8, 


Fareamt ifho had publlahad a 


c 


-24 


-07 


26 


48 


-11 


-01 


43 








03 


.'04 




book (A-27-e, d) 


H 


-Jl 


48 


33 


04 


52 


-23 


-25 








28 


*13 






P 


IS 


06 


3" 


23 


29 


-08 


24 








11 


.08' 




Avayaga nyi^ar of praseneatlons at 


C 


. OS 


-38 


29 


06 


-13 


40 


35 


34 






2*42 


1.01 




raglooal or aaelonal naaelngi 


H 


34 


37 


30 


14 


47 


15 


-28 


19 






1,90 


• 56 




(A-27-f) 


P 


-05 


29 


21 


40 


45 


-15 


24 


38 






3.34 


1*14 


10* 


Avaragt number of pragen cations at 


C 


08 


24 


49 


60 


-19 


07 


48 


45 


18 




2.22 


1*11 




scholarly colloqula (A»27-g) 


H 


^44 


09 


21 


44 


52 


19 


07 


28 


17 




1,67 


*82 






P 


40 


15 


08 


08 


11 


-11 


-01 


40 


41 




2*03 


1*05 


11, 


• 

Aliani Seala asora #2: Scholarly 


c 


76 


22 


-11 


-09 


24 


-24 


-14 


-41 


-14 


-21 — 


3*14^ 


,39 




axeallaaoa of program 


H 


30 


55 


41 


26 


05 


-13 


03 


24 


20 


-09 — 


3.26 


*40 
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-08 


17 


-06 


-11 


-14 


-18 


03 


26 


41 


31 — 


3.33 


,30 


12. 


Would adviea friend to attend 


C 


85 


66 


OS 


-08 


31 


-21 


19 


-35 


-23 


-08 — 


3.17b 


.37 




<A-l-f) 


H 


34 


58 


36 


45 


18 


08 


^2 


25 


31 


02 


3.15 


,40 






P 


42 


41 


42 


27 


29 


-46 


37 


19 


18 


23 


3,21 


.32 



*C * ^^latfy (H ^ 20 progr^s) 
H ■ Histoid (H ■ 18 progrmis) 
P ■ Psychology (N ■ 17 proorasis) 

^4 - excellent or agree strongly 
3 « good or agraa wleh resarvatlons 
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Th© wlthln-dlsclpllne correlations betwaen the variablas in 
T^le 10,4 also prMent a mixad picture. Some interpretations i 

...JudgBents that a program prepared alt™! "fxtrOTely welj" 
for work as a scholarly reaearcher, college teacher, ©r other 

i 

professional practitioner (#1) are essentially unrelated to 
any of the reported career accOTplishments , with the possible, 
exception of aurren^ employment at a Ph*D. university in ^ 
all fields (#2) and research work and journal publications by 
psychologists (#3* 5, and 7). 

...Being currently employed At a Ph.D. university (#2) is posi- 
tively related to being engagad in research aiding historians 
and psychologists (#3) and positively related to publleatrions 
and scholarly presentations in all three fields, though only 
very strongl]^ for the publication of Journal and dissertation 
articles by psychologists and the publication of books by 
historians (#4, 5, 7, 8, 9, and 10). 

.**Belng engaged primarily in research or research and teaching (#3) 
is positively but not highly related to pi&llcatlons or 
scholarly presentations in any of the fields except perhaps the 
publication of journal articles by psychologists (#5, 7, 8, 9, 
and 10). 

,.*Avirage annual Income (#6) is unrelated or negatively related 
to alflost every other variable In the table^iar all three fields. 
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• ••MosC the alwnl ac^ompllihments that are reported here are 

m 

^ asamClAlly unrelated to their own Jud^ents ^out the scholarly 
ejccellence of the progrw or their advice to a friend about 
attendmee (#11 md 12), The only exception is eurrent 
^ maiploymeilt at a. Fh.Di university i apparently these graduates 
' - think their prograos helped them obtain their eurrent posltioni^^ 
and they weuld recoiDmend the programs to friends with similar 
/^interests. 

Correlates of Career Accomplishments 

Measures of alwani accomplishment do not appear to be very 
highly related to one another. Are they more highly related to 
program characteristics such as reputation^ faculty research 
activity s student views about the enviroiment for learning i or 
overail satisfaction with the progrM by current students? 
T^le 10.5 presents some of these corralAions^ 

Again we sea soto congruence be^een the research preparation 
and involvraent of alimni and the research emphasis of their 
graduate programs » as represented by correlations of Itms #2| 3, 
and 4 with peer rating of the faculty quality and level of faculty 
research activity* But there is very little relationship betwera 
any of the four progrra characteristics and the scholarly produce 
tivlty of alinmii as reported by the number of Journal publications 
or ptesentations at regional or nAtional professionfl meetings 
(#6 and 7). The number of publications by altmni also are not highly 

related to the publication productivity of the department faculty 




L "Irjiiiiy good" eariier 
prepariilgn 

I Good prepiration for drier 
§8 ichollfly raiearcher 



3i Currint position at Ph.D. 
ynlviriity 



19 JO 31 35 6i 



, Reiaarch or reieirch ^ taaching 

ictivicy 23 65 iO 

'21 li=31 



5, Averaga annual 
u. kmm mki of joinT 



-25 §7 52 
43 -13 -50 



n n II 
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[h Seili 
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# 




involvement and productivity are unrelatad or only weakly related 
to the environment or learning and swudent overall satisfaction 
with the progrma* Average Income is not related or negatively 
related to all four program indicators (#5) . Only with the last 
variable t aluami-rated scholar ly excellence of the progrMi is 
there a consistently high positive relationship with program 
reputation, faculty research activity * and student satisfaction. 

The lack of any consistent pattern in these relationships 
suggests that the post degree activities and accoffipllshnients of 
a program's roccnc gi:aa;:aL'iS are influenced by many factors other 
than the graduate program itself t and that any effort to evaluate 
the quality of programs by evaluating the emplo^ent settings and 
professional accomplishments of graduates would need to be much 
mnr^ mn-nhi ^ri rnr^r^ th^^n fhe flnrsrnRch taken In this study a As we 
saw in Table 10t4s even the graduates' judgiiionts about the scholarly 
excellence of a program fail to predict actlviules that aye generally 
considered to reflect scholarly achievement and recognition, such 
as a re^^^r academic appointment at a Ph*D* university or scholarly 
articles published in prufessional journals # 

In addition to information about alunml emplo^ent anu profes^ 
sional activities* it could be helpful for departments to Imow more 
about the value of various graduate school experiences as preparation 
for different kinds of work. Table 10,6 lists several items of this 
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Tabl# 10.6 

Aluml-Rated Valtie of Graduate Experle.nceii as P-reparatlon for 
PrtSMC Work (Averages of Progrm Percsncagma) 



Chemistry History Psychology 

Biperimioe of. vary graat valua 

M praparatim for prasrat work. H M L* H L* H M L* 



1. 


Requirec!! courses In dept. (A— 6— a) 


19 


24 


28 


28 . 


31 


13 


34 


32 


17 


2. 


Eleetlve courses in dept.' (A-6-b) 


20 


27 


29 


43 


46 


43 


42 


33 


29 




ASBQCiadon wicn your najor 
professor (A-6-d) 


63 


61 


67 


63 


68 s 


62 


56 


63 


63 


4. 


AssoclaClon with other professors . 
(A-6-e) 


22, 


25 


26 


45 


34 


32 


43 


36 


23 


5. 


Association with fellow graduate 
students (A-6-'f) 


48 


36 


29 


36 


34 


37 


50 


46 


39 


6. 


The departnent's standards of 
excellence for work In the field 
(A-6-h) 


60 


34 


24 


64 


53 


36 


68 


60 


42 


7. 


Cultural and social life of the 
university (A-6-i) 


28 


16 


11 


32 


23 


13 


31 


19 


15 


8. 


Technical skiffs learned In course 
or research work (A-6-J } 


57 


52 


68 


44 


34 


36 


63 


59 


<56 


9. 


Ibowledge gained in course or 
ceaeareh work (A-6-k) 


58 


50 


59 


60 


63 


54 


67 


56 


49 


Prepared "excr^iely well" for work In: 








/ 








* 




10. 


Sehelariy research (F-/4-a) 


74 


48 


37 




. 53 


.45 


76 


69 


59 




College teaching (F-4-h) 


25 


31 


2 


16 


35 


45 


22 


22 ' 


26 


12. 


Other professional practice 
(f-4-c) 


10 


7 


9 


4 


3 


0 


2Q 


24 


37 



*Saa footnota^ Tabla 10, 1« 
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sort 




to dlselpllne antf^tW** v^f^Q 

^ulatlons BwmAt±z& responses by aliaml In all kinds of pasltisnsi 
for l^eal pTO^x&m review, It would probably be more helpful to 
sort responses according to type of work* 

^ From the items in Table 10.6, it appears that ^^«oclatlon with 
a major professor (#3) and technl::al skills or knowledge gainad In 
course or research work (#8 and 9) are perceived by dlmml from all 
kinds of programs to be of greatest value or use In preparing thm 

■ 

for their present work* Elective courses and association with 

other than the major professor 01 and 4) apparently are more 

cosEon, and more impprtantj in history and psychology than In 

chemistry. Importance assigned to the denartment's standards of 

excellence for work in the field (#6) vary according to progwm 

reputatlonal levels as do departmental and general university 

Interpersor il experiences (^>5 and 7). Reports of extremely , good 

1 

preparation for work in one of three major career paths (#10-12) 
closely parallel the differences in program emphasis that were reported 
in Chapter 4. 

^ The retfi Uu reported so far in this chapter are consistent with 

the notion chat most doctoral degrea reciplentSi almost regardless of 
the quality of the programs that grant their degiees, adapt to ^he demands 
and expectati. is of the emplaynient situation. Three to five years af t€r 
receiving the degrees they ref le t the associations and expectations 
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of their graduate progr^u only If the employmaat setting has 

inebiirag^ syah^tfiavlor. For sxai^l(%^ a hMvy teaahtog lomfl 

vlil absorb nost of the mamTgy of a new faculty a^d»sr^ ragardlsss 

of the aphaals plaesd on training for rMearch and publleatlon - 

hf his or har graduate program, A survey of ^^loyers mlg^t 

uncover some differences In Job perforwnce related to the program 

of training, but it seems more likely that much of , this variation 

Is rCTOved by the self-sorting of career aspirants when they select a 

graduate training progrm to attend ^ and then by the jobs they seek, A 

relationship between eharacteristics of the progrra of graduate training 

_ * i 

and alumni career performance might be mere apparent if alumni were 
carefully sorted according to their current raployment set things 
and whether or not they had received specific training for the 
, activities €Apected In that setting. However p getting a sufficient 
nusAer of responses from any me. group to reliably serve as a 
p^ograi^level indicator of performance would be a serious problra 
for many programa. ^ 

In mwy respects i the measures developed and used 1a this study 
produced more questions than answers | they are difficult to Interpret 
since they do not appear to be r^ated In uy conaistrat way to^ the 
other program s^asures that were obtained from/faculty mmbers, 
curr@it students » or department records « There are at least three 
possible explmiations for these results. (1) The data are Ijiadequate; 
th^y.are not c^^lete enough to give us' an accurate plctiure of almil. 
acco^llshamts and opinions about the progrms from which they 
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reel 
had 

beuavloP ^^^^^^ 

that data Inadequacy is a sufficient explanation of the results. 
(2) Munmi of a progrm have a vested interest In fflaklng that progrm 
appear as 'strong as possible and, therefore, teni to view their 
eacparienees and tr&lnlng through rose-colored glasses This WQuld * 
tend to obscure differences between programs in different feputational 
groupings. However, although there may be some strain toward reduction 
of dissonance In the alimil responses , there is no apparent reason ) 
why this should b& any more true of aluml returris than of returns 
from current faculty members and students* (3) Almmi Jobs Bnd 
accomplishments are determined by many factors, only one of which 
is the particular prngram of graduate training. We have seen that 
probably the prestige* of nhe program that grants the degree Is 
helpful In obtaining i regular academic appointment at a PhpD-- 
granting univet^xLy, lu^l piu^iTtiai^ wi th v&ry ntrong 

emphasis on training reaearchers du have more graduates who are 
employed in research work* But whether the person is employed In a ^ 
^ob highly related to his or her field oi graduate study, feels 
underemployed J publishes scholarly articles or books, presents papers 
at meetings or colloquia, or fets a feeling of worttwhlle accomplish- 
ment from the work Is largely independent of specilf ic characteristics 
ot the graduate program^ This view suggests that the alurol data 
could be Interpreted to mean that some* but not alii of, the graduates 
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of mlaest mwrnsf ptogtm in thm mtu^ are doing the kinds of thtngs 
that iiQuld be omsldarad to reflect excellence, and that careet 
'paths mvm detenilnad less by specific graduate school experiences 
thwi by othM faators (Including the initial selection of a graduate 
prog^an'ta attend). Alt^mi responses may also rr^^ .jct less ^ "^uence 
froii the pervasive notion of a hierarchical arrangement among 
graduate pvo^rmmm than do responses from faculty members and stuaentH, 

If thasr conclusions are accurate , the relevance oi alumni 
maasursn to program evaluation needs fu^th^r study* There may not be 
th% automatic connection' that usually is assumed. Such measures wouid 
naed to be developed and interpreted with great care for they may 
challenge a humber of traditional views about the characteristics c 
programa th^t are most important for educa*^ional exuc^lencc. 

^Umni QpinioaB for Frogrm Improvroent 

A second reason for surveying recent alumni as part of pr^^gram 
reviw was to solicit their opinions about particular aspects of 
tha pragram and the training they received* Some itms from the 
atud^t quastiofinaire were repeated in the questionnaire sent to 
aiumiii, Inelttdtog a nt^er of the items that were combined t ) foun le 
syalii seores reported In enrller sections of this report and 
daatrll^ed is detail in Appendix F, These items were tabulatid as 
put of the^aedbaek reports and copies sent to participating 
pTQ^mm* Saveral department chairmen eKpressed particuj ar interest 
tn^tha aliaml qoemanti about the program^ 



Xhe alunmi res: -naes to ito'-.H ^oricernir - ^-epr rti.ie: _al 
fun. ioning were correlated vlch s:-.denL rv^^^r n^es n 1 jenLic?iI 
items to get sore indicaLicHi ui T])t'.r itival of agreeraent* Noc 
surpt Islngly , tl.ere was a centiiitillv h h dos . ve association--- 
visually in the range cf ,5 to * 7--t it some considerably lower 
correlations as well^ Ther^was a ^^er hi%h level of agreement 
about the ability of the faculty and tlie f^verall excellence cf the 
program in all three disciplines to .8). Agreement was firmly 

\ 

. tudents i humanenes? of the env ronmentj or :ideq'". cy ?f the cut ^icul' 
and academic procedures « There vas least agreement on ' 3ms that 
may wall raflert different expeviences and conditions ^ such as 
student support for one another or faculty concern for students, 

btany alumi comnented that they had little first-hand informa* 
tlon about re ent changes in the program and often added that they 
had been gone from the cmpus since the middle or late 1960a, 
Therefore, very real changes w \ have taken places in addition to 
differences in the perceptions of the two groups. For this reason, 
it seems likely that the opinions of currant students will be much 
mora useful in assessing the program procedures, environment g and 
curriculum that exist at any given time, and that it will not be 
very profitable to solicit opinions about these same chai'^Anteristics 
*"rom alumni. One exception might arise if a departmant ^^antad to 
document the effect of program changes on the attitudes and opinions 
of participants, but this would call for a particular set of ques- 
tions to get at the Issues of interest to that department* 



f 

Aluniri ab ^ We^r^ ^ -y^ ' ^^ , -xr r^-. e 'h 

hfeipfuiness Br. :e--.:. pr- : ■ f- ' ' ' >s _ w-t.r 

they nrrp^etea L::t_^r ucc :rai sL.. 'ie.^. A r:i:ir.: > a ha^r or zn^- 

prestige levels s^id that: .he s.;. ; ^ n.e _ .ndiv^:dual pr^fesscrs 
and ier^^^^s ient directly to urns er.ivei emp'cvers were "v y 
helnfui.'* The aepar tLnent ^ s fDrmai or informal placement effects wera 
judged ' v r% helpful^' :?v IS-: 20^. of tliL graduates of mo:^^ prog- .is. 
up en i rigs xi&tt!u wiuu c* ^ t ^ ^ > i. ^^o^. ^ r^^*.- 

.0 hisrar^:nj p 5ychvlrgi - 1? r>^i- -hpn^.-rs H 0^ 1? vs. 7%^^; 

psychologists, particularly, rarely found tae university pj icement 
office of rauch help (3% vs. 7% to 11% ^Very helpful^-). Ic is 
Interasting that the alumni responses did not demonstrate differances 
riiftcd to departmental peer ratings as did the judgmants by current 
students and faculty that w^re raported in Tabla 9,4, 

In this chapter we have shown that the alumni who rasponded 
to our questioui>Airas ware qui.a similar in several ways to currantly 
enrolled advancaa -aduate students in the oame pro£ram8 and, tharefore, 
probably represented a reasonable cross-section of the 1970^1972 
graduates of these programs even though the response rate was rather 
low. ;he alumni were in a variety of positions and reported a variety 
of accomplishments that one would expect to be ralated to tue 
academic training they received, scholarly publications, prasantatlons 
at professional meetings, and the like, Howcivar, i xcapt that 
there is soma association betwaati the research emphasis of a t^rogram 
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and the employmen:. of graduate:: ^--ise^ircK ^c-z>^ i' ..i ? ccc-p 1 : sh:;.tn l 
saeine4 largely unrelated '\0r r^j^^^^a in ^:ry .^unirl^^. ,ays that 
wa do :.it yet understand) t^-; pr-^rair; ^ita^-is m the .:'th>=r niajor av^-^.H 
of facuAty training and produc t i vi tv , student abilitic:^, the cnvlr .r^ 
ment^^ available resources, the curriculum, and reputation* 

feedback from aluioni concarr.ing iheir emplovment , otht.^ 
profassi^nal activities, and opinions about the graduate program 
in 7 lation'to the wurld of work are much more important than their 
reminiscences abou^ tneir stuaent days. We have discussed some 
problems with tlie in.terpr^LaLioii ui the -.e kinds of uaca as compara- 
tive indicators of program outcomes. But this does not reduce the 

department's self^study for program improvement anu internal 
univarBlty review* Items such as t^.ose reported earlier in this 
chapter and in Chapter 9 would be equally atjproDriate for self-^study 
purposes . 



This Study gUeW uUt ol ^4 i dbwuL Llie 1 -.Ht ^ L d C i OnS Ot ptftvl 

ratings of ^-'culty quality^^of ten re '■erred to as ''reputat tonal 
r at ings'*-^as t^^ crly widelv acknowladged indicate i c:. dsctcral 
program quality. Because of a nurriber of re. ^.rvat ions about the 
meaning and utility jf peer ratings, tlie Co- ncil of Graduate Schools 
in the United States ioined Educati nal Testing Service in an 
^^ffort to determine the feasibilitv --^i collcc*:iTi"^ morQ detailed 
information about the qualiLy of t'.octorp.l prograrus , with the eventual 

evaluated* 

The study reported here used, as i starting point, infoTOation 
gathered in a naticnal ^^^rvcy of graduate school deans conducted 
several years ago (Clarke 1973). ThaL survey polled graduate deans 
regarding the criteria they thought were important In making Judg- 
ments about doctoral program quality p as well as their Impressions 
about the merits of va' tous possible sources of information about 
these cr teria. "^om a list of almost three hundred posslbla 
doctoral program L.haL i^Ler ti^.6 * ^hc tlilrty which received the 
highest endorsenien^ as reasonable Indicators of program quality 
are llstad in Table 1,1 xn the first chapter of this reporr. 

In an effort to examine the feasibility of r jing mp^y of these 
characteristics as IndicatorB ^ nuality- in .u<^toral educationi 
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information w^s gathered. "-r;^^ M onra i ref^ , f'rom h -uJents , alu-nl . 

faculty, and cru^lrrr^vn ir i: :\ r . • rr- - iv -iszr- and 
psychclogy, nnd ^5 progr^-- .r laaic questions guided 

the collectinn an^ a^qivti^e n^' rhoaji. iat = H) Could che various 
dimensions be mr ^isured reliably and within reasorabie boundaries of 
cD-t and inconvenience? (2) If , wnula the data provide useful 
infonnation , information fnat would ir. rove judgnients that would 
otherwise be made ab.jut programs o . the Dasis of their aize, reputa- 
cion, or OLner axreauy civdLx.xnL^ lh lu i imAL luh ^ 

Eviu*.^rict^ i^U ^..^ I- ..^ ^wo qu^srion^ i:i prci^cnted in t:ie 

previous chapter^, . i wl ich l\.c- rr^anv possible indicators of program 
rinilifv prn -trH in to i x catettories having to uo with the 

quality of (1) faculty, (3) s*-'idenrs, (3) resources, (4) environment * 
(5) alumni, and '6 J program academic offerings and procedures. The 
purpose of this chapter is to review some of the more important 
findings of the stiidy with resect to these areaB of doctoral 
quality, and to discuss the impl4eations of these findings for 

future efforts to assess quality in doctoral programs. A very 

/ 

brief overview of findings with regard to the many possible 
indicators of quality is given in Table 11.1, and a slightly more 
detailed discussion of these indicators follows. Before turning 
to these areas of program qualitv, however, wf- ill consider the 
question f^f prngraTTi piirnn^^f^^^, , 



Characteristic 



Faculty ; 

1, Pircentaga of faculty with 
doctorate 

2, Percentage of laculty with 
doctorate froin h 'ghl}' nted 
progrfflni 

it Peer ratings of qualitv of 
facultv 



k-. Stiident-rated quality 



5. Faculty-reported mm 
articles/book reviews in 



L Faculty-reported books 



published 



7* Publlraflon cltarlon Indpx 



8, Salarlgs 



7< AeaeaiUi atki-v^Lji xuuc, 



ERIC 



COMient 



Verv little variation across progranis; not sufficientry 
discrMnaLing to be very uiiful as index of quality. 

Also not v,ry discrliinatlng acrosa prograis in cheiistry 
and history, More variations- (and therefore potentially 
more useful) in piychology. 

Very houiugeneous (high agraisent aniong rater i) and f table 
over time; highly correlated with riieirch uphasis, size, 
publications, and the like; not correlited wich indicitori ^ 
of the quality of the environiint for learnin|, Requires n 

r.nllection of ratings data*froffl facuUv In other iinivf rsitieR' ^ 

Data reasonably to |it; highly correlited with 

eeve il nieasures of the quality of the leamini environiint; 

not highly cofRlated with pter ratings . 

Wide varicitvjn across progrms; fairly highly correlated 
with peer ratings; probably more useful index than lean 

in chiiiij; ;y and piycholoiy, 

Not reported ''Iti ?ufficient f'^equency. to bs useful In 
chiilstry ind psychology , bur. a Eore usiful Indicator 
in history than articha publishsd, 

Very hi|hly correlated with publication counti; riquirii 
great diil of tilt and if fort to gither nicissir^ dita. 

Niither conciptuiUy nor liplricilly difinsibii as pnuini 
indicator of quality, 



perfommci of a departiint s ficiiltyi 



(continugd) 
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Characteristic 



Studen£s : 

an index of snectr'ity) 



11. Qualitv of underi aduaEe 
institutions 

12. Student self-riported under- 
graduate fiide averagR 

13. Student coinitmint, 
uiotlvation 

14. Quality of dissertatioris ' 



1!). Studenf ^''trltlri (nr n-p 
of ABD's) 



^ = M p * " * 

16. Actuil physical resource 
leasures (e.g., books in 
library, .'ab faciliEics, etc.,, 

n, Ficulty self-ratings of 
adiquacy of facilities 
^library, labs, etc.) 

IBi financial and other program 
disctlption data 



^19. Ovetall institutional finan- 
-ERLC stabri^;.' indicitor 



in chapter 



highly related to program research enphisis and peer 
ratings but data obtained in this study were too Incoinplete 
to pimit confident conclusions , 

Difficult :o get a|reeient reprdi^, meaning of undar- 
graduate institution quality; variaDle not exiiined in 
this study, 

6 Fairly highly corrdated with graduate faculty appriisals 
of scholarly ability, coniitffliL , and motivation. 

6 Measurid here by faculty ratings ;hi|hly correlated with 
ratings of schQlarly ability and coBunication ikills. 

b Would require externil pmel of ritirs, perhaps also som 
inforaation regardin| nonis for such ratlnRS across my 
piOiraiQs; variable was not included in this study, 

^ Difficult infonatiOTi to collect; data riported by deoirt- 
inents were frequently labelled as guessis; would seia to 
be iiportant charactiristlCj but this study could nut 
analyze in any detail. 



7 Very difficult Information to latheii ispeciiUy across 
large nuiber of proifais; not usid in this study. 

* 

1 Data lasy to collect and an quite riliibli; positively 
correlatid with piir fatingi progrw iiphisis on 
usearch in all thrgi fields; also correlitid yith 
progrfli Instructional quility In chffllstiy and psycholofy. . 

7 Extreiily d.. ■^eult to obtain rillibli infonniEion acrosi 
pro|rp§; lany analyges Intffldid with thise data vici not 

InforMtlon, | 

? 'Not exaalnid In this study, 2u 

(eetitlsuid) 



ChiracEerHEic 



Discussed 
in chaDter 



lavironMnt: 



20i Student repoiLb abuuL vdrlous 
asptctp of prograi envirorr 
aentj luch as faculty concirn 
for students j cr petitiveness 
ef itudintSj etc. 

21. Faculty reporci about various 
ispicti of invironientj such 
as pregriffl leidersiiipj rela' 
tlens with other faculty in 
dept., etc. 



Coment 



'^„t,. ,„p,, v., ,,ffiVs/>t' jnfi -sva ''fll-isMp' pn'^Tpl qfpfl with 

wyt-.' ^^--J ^ • =^iLi^ .... . 

quality of teaching; not related to pier ratings and 
other traditicnali riiearch-orianted indicators cf 
prograi quality. 

Data easy to collect a-^d are reliable; relationihlp with 
orher traditional indicacors of propaH quail ^ (e.g., 
peer ratings, prograi eiphasis on research) varies by 

difeciDlinc. 



Proeadures 
One or nore efflcienc;. indexes, 
such as costs per student 
cfeau nour ot tu^L pei uegLct 
awarded 

23 Student ratings of various 
progran contents and proce- 
dures, such as flexibility of 
requirefflentSj assistantship 

24. Average time tc degree 



AccL-rate data across pro|rains extreinely difficult to 
collect. Not treated in this it 



ThesQ data can oi obtained fairly iasily and, as leasured 
.1 this study, are quite reliable, They provide a useful 
"process" indicator that can be coipared across ;'rogran!S. 



6 Not. included in this study. 



(contlnuid) 
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Characterise -^^ in chapter Cofflnen[ 



,^\antuilj long'tern, pro- 10 Though eventual, long-tiiij criteria wguld be desirable, 

fasilonai accoopl .hffients obtaining follow-up data froffl those who earned their degree 

(iig., awards J prgfassional ' iflori than a few years ago would be extremely dlfflculti 

iociety officerships, etc j Nor. atteptad in this study. 

26i Professional parforniarce 10 Parfonance inforMtion of sore fecint iraduatas is aasiiir 

(a*g*j f^ablications, job to obtain^ but the short tlss lapsi sinc^ tht degree provides 

riUled to craining) of little opportiaity for stable roccird of accosplishienti 
ricinE digrae recipients 

Ih Ratings by recent :dunini 10 Alumni ratings data are reliable; overall progria quality 

of various pregra.^ ' ratings of aliMii highly correlated with self-ratings of 

characteristic^ faculty and peer ratings. 



H 



i 
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The Role o l: P"- Qgraffi P ur poses 

One of the major objective.^ of this study was to gauge the 
e^ctent to which it might be pcssible co develoi: different indicatora 
of quality for doctoral progr ^^fferent program pur^o.^es 

and amphaaes- One of the shortcomings of previous efforts to assess 
quality in graduate education seemed to be he failure to recognize 
that the criteria usually employed to maaaure qual ty-^^e.g. , reputa- 
tional ratings s publlcacions the faculty , and the li^.fi—were 
criteria that wer ^ primary relevance to progr^s empliajizing the 
preparation of resaArchtrs and vere less appropT ate to use with 
r^^ograms de Igned to prep -achers or other practitioners. It 

was for this rear^n that ^^tuc nt and faculty respondents were asked 
to indicate the degree or imporcance chey thought their departiflent 
assigned to three different purposes : preparing scholars and/or 
researchera, preparing college teachers^ and preparing other 
practltionars . The lnten?:ion was to use reeponses to these Ilmis 
to sort departments into ditferent groups according to prograp 

A^^^ frnTT^ phoRe that nlaeed malor emphasis on preparing 
teftchers would be examined differently from th« data concerning thoae that 

It turns out that our logic was exceeded by our naiveta, km J 
already carefully documented in Chapter 3» wh Uh presents the data 
relevant to this question in considerable detail, we ware not 
prepared for the consistently strong and pervasive Ihfluence of 
the research and scholarship model at the doctoral level, and the 
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corrMpondltigly slight emphasis given to preparing teachers and 
pMctitlottMS. Though the idea q9 fitting assMSsmanc Indicators' 
to the purposes of doctoral programs sounds reasonabls in theory , 

m practice its j.mpact muLt^d by the aimpie fact that the over- 
whelming majority of doctoral progrMS in the country appear to 
IdBntify the preparation of researchers imd/or scholars as their 
primary goal. This wa^ certainly true for the gtaduate departMnts 
In our study, even though we intentionally included a nmAer of 
programs in our sample that were not mong the elite in their 
respective fields* 

Thus, our discuBsion of indicators of quality will *e seen to 
have been influenced strongly by the traditional research and 
scholarship model* We do not ipiore other progrMi purposes and, 
rn he Rtire, a number of possible quality indicators are identified 
and discussed which would be appropriate for doctoral programs 
legardless of their primary emphasis* But, on balance, the indicators 
are ones which give recognition to the strong orientation of mnst 
docforai programs toward the preparation of researchers and scholars* 

Peer Ra tin gs of Faculty Quality 

Data presented in this report raise^ serious questions about 
ujing peer ratings of faculty quality as the sole criterion 
of doctoral program quality* Besides various ^shortcomings having 
to do with the rationale md utility of peer ratings (discussed 
in Chapter 2), the analyses reported here suggest that, as the 
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slngla deftoer of prugtm quality, they have anortcomlngs In an 
emplrlMl sMsa m well. Th©ugh peer ratings art in general 
associated with i uMrous other traditional indicate rs of doctoral 
progrdm quallty"that is, wl\^m examiueJ aeros^ a group of do noral 
programa, pear ratings are definitely higher MBOng those with hl^^ 
publication rates 9 level of research activity i qualiny of reso'ar.'.es , 
rad the llke"there are cases in which individual departTOnts a^e 
a clear except Ion to this general tendency. The faculty in 
doctoral progrras have very high publication ratefi «id reseirch 
activity records, for example s but did not reeei high peer ratinsfi 
regardless of the general tendeucy across a lai nu^er^ " prograsis 
This night Mm that .ho^e rating tftlB programs were able to ma^ ■ the 
sometimes difficult distln^^tiutrrt^pen publication quantity and 
publication quality. If so^Hhe fack o£ correspondence between 
publlcati^on rates md peer ratings for an individual departmeut may 
simply naan that publication rate is ..at mi accurate measure of 
quality for the deparcment In question, alternative Interpreta- 
tion! however, is that it la the peer ratinb that is inaccurate 
since It is a ffisasure i^ich* in effect, has nor kept up with the 
"real" activities and characteristics of the department* 

Tkiere Is no way to determine fr^ the' data reported he^e which 
of these two explraatlons Is Mre reasonable. But su^ interpretive 
difficulties appear fairly often ^en eKamlnlng'peer ratings data 
for Individual departments, aad when there Is such a discrepancy 
between peer ratings inf@jcma^4^ and other departTOntal indices 



26-/ 



11.10 



of quality i such a discrepancy shDuld mmT^^ as a ''flag" to warn 
the evaluatoi: of the difficulties of making judgmants about 
prograoi quality on the basis of any single IndlcatOT. 

A Curther limitation at peer ratings is that they are not 
ralatad to several interi^al ludlcatora of progran quality s such as 
quality of teaching or quality of the envlronfflent* Favorable 
student opinions about these two characteristics are foimd Just 
mm often at departments with high peer ratings as they are at 
dspartmants with low peer ratings* Thus, over a gfoup of departments, 



departmantal features » 

" These findings when combined with the logical shortcomings 
of peer ratings discussed in the first chapter, suggest that peer 
ratings are not sufficient as a single criterion of quality in 
doctoral progrMas* On the other hand, they clearly are appropriate 
as one of several indlcal jrs of faculty quality. They are 
generaliy stable over time, yet the ratings of some programs 
do change; they are reliable ^ In the sense thmt there Is a 
high level oi agreement among diffarent rioters of the same 
progrwa; and they are "valid" xn the sense that they are 
correlated in expected ways with a variety of wther program 
characteristics (see Chapter 4), Thus, as o^e indicator of the 
quality of a Sectoral program's faculty — faculty quality, that is, 
in the reiearch and scholarship sense—peer ratings are clearly 
appropriate and usefi'l. 



peer ratings data are nof at all revealing abou^ these important 
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other Ipdleators of the Quality of the Faculty ^ 

Three othsr Indicators of faculty quality ware Identified mm 
ones that are generally available and can be reliably assessed* 
These were th^ student-rated quality of teaching, the average n^pber 
of published articles and book reviews per menbar of the faculty 
In the last three years, and the faculty research activity, (A 
detailed discussion of these variables and their characteristics 
Is presented In Chapter 5*) The latter two characteris^cs tend 
to be positively correlated with each other and with ^eer ratings, 
that Is, departments with a high record of publication productivity 
also tend to ba departmciiits with high reputational ratings and 
much research activity* Nevertheless ^ there are enough exceptions 



to chac tendency to warrant asaassmant of each of these character is tics 
Student-^ra^ed quality of teaching, on the other hand, appears to 
be unrelated to the department's research excellence, as defined by 
the other three indicators. 

Assessment of a faculty 'b reaearch produc'tivlty by paylnii 
particular attention tu the quality of its publications as well as 
the .quantity Is an appToach suggested by some observers. However, 
previous efforts to assess productivity by assigning weights to 
publlcaLlons of different quality usually end up with essentially 
the eame results as counts of , their number (Jauch and Glueck, 1975) 
Furthermore, such procedures are difficult to carry out and have other 
problems that cannot be easily solved (e.g., the great' diversity of 
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Opinion abaut which journals are the b^st ones In a given field) • 
Wot chess reasons a slmpla average of publications during a recant 
period ^s' probably more realistic^ at least when one's purpose is to 
ideatify an Indleator of progrra quality that Is useful aoross a 
group of departments or pTogrmss. It should be mphasiied that whan 
it eomes to individual program or Individual faculty mCTber evaluation 
however^ careful attention to publication quality as well as quantity 
is absolutely essential » 

Besides faculty research performance and teaching skills, two 
other characteristics often thought to be important to faculfy 
quality have to do with faculty backgroundi nuelyi highest de||ree 
earned » and quality ratings of the program from which the doctorate 
was earned p Though 'we obtained Information regarding these background 
characteristics J neither proved to be very helpful In distinguishing 
among programs » There was practically no variation am^^ng departments 
with t aspect to highest degree earned p since virtually all graduate 
faculty in all three fields in this study have doctorates. Thr 
quality of the program from which the doctorate was earned (as 
measured by peer ratings of those progrwi) was aommfhat more 
promising » since there was more variation among departments on this 
variable (especially in psychology), and it is sr^ewhat related to 
other measures of program quality. Nevertheless, it serves mainly 
to dociment the fact that a very large proportion of doctoral 
program faculty earned their doctorates i^'^m high prestige programs. 
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QiialltY of Studgnte 

Two prOBiatng maBures of Btudant quality mrm thm students' 
imdergraduatt gradas and the faculty-rated coTOltment and motivation 
of these acudents^ Beth of these indicators are readily available* 
can be maaaured reliably and with minor inconveniences and yield 
algnifScant differenees between departments* 

A meaaure of quality of students that did not prove to be 
available from the records of a number of departments was Graduate 
Record Examinations scores. Though this would sera to be a "natural" 
and convenient indicator to use, the tests were not required by all 
departments and some others were unable to provide a^iirate average 
test scores for their enteriug students* Thus* until departaents 
keep better records of the GRE scores submitted by students, this 
Indicator of entering student quality cannot be expected to be broadly 
available fnr comparison between programs. 

It is important to note that neither of these indicators 
(undergraduate grades or faculty-rated commitment and motivation) 
has to do with the quality of students leaving the doctoral program. 
Two atrategles for measuring quality of students after completion 
of their doctoral training are available. One is examiiie the 
i^entual performance of the program's alumni; we attcimpted to 
obtain data of this kind and will discuss the re^ ^ t ts later. A 
second stratjgy would be to examine the quality of a program 'j 
graduates at the time they earn tneir degree, av, Jot exampJ^., 
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through an evaluation of Ph,D, dissertations from the departtn^nt, 
Unfottunately , this procedure would be expensive and time-consuming 
to aetabllsh on a regular bails, though 1*L might be an Integral part 
of a p#riddio program evaluation procedure that leans heavily on 
mnv te^^iew. In any events given the constraints of limiting our 
indicators to those which could be provided by students, faculty, 
alwnit ov program admtnlstratorg , such program review ^dlcators 
were not Included in our study. 

Quality^ q£ Resources 

The adequacy of a department's resources can be assessed by 
either measuring the resource in question (e»g#» the books in the 
library, the laboratory space available) or by sSRing for people's 
'Opinions about the adequac/ of the rrmoarces* Ouir analysis Is^rus 
us to the coftclusion that^ since faculty tatings of resourcts are 
easier to collect (than actual book counr i, for exajcpla) ^ are 
reliable and ara related to both thf: resaarcb and educational 
functions of graduate dep.rtnenLSi chey would probably be better 
meafeures oi thu rfesoutcu Jimun^iuu, Thfc r^Liugs of the f acu3 "y 
^xffiost cerL^Jnly would be more useful and far e^eie?; to obtain than i 
".he expensive and tinie^consuming inethod of att.*Dmpling to cbf^In 
cpe-iflc, standardized measures of all relevant resources and 
la.^iities. C ;c '"'^^npter 7.) 
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Quality of Environment 

Ths quality of the "environmsnt" la a dimension of graduate 
tduaatlon that is aeldom acknowledged « In this study student and 
faculty perceptions of- their environaent were measured by meanc of 
questiohnalras« Studiints were asked about such aspects of the 
environment as faculty access ibility and treatment of students , the 
-t^'tent to which competition among students semed to be encouraged ^ 
the quality of their relationships with other students. Faculty 
m^bers were asked about their relationships with other mmbers of 
the faculty^ their opportunities to influence departmental decision 
and the like^ 

From thaaa data^ three useful measures of the department 
environment emerged i first, the nature of the learning environment 
(as reported by students) » second* faculty concern for students 
(also as reported by studenti) ^ and thirdp the nature of the work 
environment for faculty « Each of these measures sewed to provide 
provocative and useful information abou= student and faculty 
enviromnents in all three disciplines. 

The quality of the student environment ^ which s aems to be 
primarily an index of the "humaneness" of the department, was not 
related to indicator^ such as peer ratings and publications in 
chemistry and psychology. In other words » humane environments were 
found approximately as often among prestigious departments as 
lesser Imown ones in these two fields. In history departments, on 
J 

H\7 



11.16 



the other hand, favorable student environments were found less 
o£tm in departnents with high peer ratings and strong research 
orlratatlons . 

The quality of the faculty environment, however, was found to 
be associated with such traditional characteristics as peer ratlnis. 
program eBphasls on research and scholarship, md the like, sug- 
gesting that what is important to the faculty Is uot always what 
Is liiiportant to graduate students. (For further details regarding 
the nature of the envlronnent indicators, see Chapter 8. For a 
discussion of the utility of environment measures, see the section 
entitled "Internal versus External Indicators of Quality" later in 
this chapter.) 

Academic Of ferlngs 

Like the nature of thp academic environment, opinions about the 
quality of specific components of the totality of the graduate ^ 
experience are seldom sought. We asked students and aluiiail many 
questions about the vat ous specific aspects of their graduate 
training, and four groups of items turned out to be ones worth 
recommendlngi (1) those dealing with overall - cudent satisfaction 
with the doctoral program, (2) student ratings of the curriculim, 
(3) student ratlnga of their assistantship experiences, and (4) alumni 
ratings of their diasertation experiences. Each of these measures of 
the academic offerings of the program tend to be correlated with the 
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student MivirOTraent measures d#serlbed earlier | that lS| departmant? 
In whlah there Is a high axprassion of student gatlsfactim with the 
Assistantshlp eacperlenct also tend to be departments In which stu* 
dents report a favorable envlrotuient for learnings and so on. But 
the relationships between the ratings of the environment «id the 
ratings of the aeademie offerings are still suffieiently dlstinet 
that their separate assessment is Justified. 

Aluml Ferformmee 

As mentioned earlier » an analysis of the perforaance of the 
alusnii of a given departmant \m frequently cited as one of the moat 
sensible ways to attempt to measure the educational outcomes of a 
doctor 4i_progr am. According to this argijaaent, the most effective 
departiaents would be those whose alumni achieved the greatest 
distlnetlon In their postdoctoral years. 

In order to obtain feedback information from alwonii who hatf*^ 
been in j ^ j long enough to have some record of accomplishment but 
who also had graduated recently enough to have first-hand knwledga 
about the doctoral program from which they had graduated, informa- 
tion was sought from those who had earned their doctorates from 
three to five years ago (i«e«p those who graduated between 1970 
and 1972) « One of the first problems was getting reliable data. 
Only 46% of the almmi we contacted returned usable ^data* Though 
the respondents do not differ in any consistent way from students 
currently enrolled in those same graduata programs » we naturally 
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mimt hm concerned about the extent to which the alumni data enable 
us to be aonfident about the experiences of all alumni in he 
ample. More Important is the lesson in this for the future i It 
im elear that most efforts to obtain information from program 
alwml will also encounter the problem oi response ratef and the 
eorresponding concerns about the generaliiablllty of the Information 
obtained. ^ « 

With this caveat about response rates in mlnd^ it would appear 
that four alumni behaviors are potentially useful as Indicators of 
program quality i (1) altmni-rated quality of their career preparation, 
(2) the percent of the alumni who report that their purrent job Is 
highly related to their graduate training, (3) the ppblleatlon rates 
of alumni, and (4) the member of alumni presentatio(is at regional or 
national meetings. Nont^ of these four indicators of alu^l 

accomplishment are cone is tent ly correlated with other characteriutlcs 

1 

of graduate programs, especially those having to do with faculty 
training, pi Qductivlty , and reputational rating. 

It is worth noting that the first two alumni Indicators are 
appropriate as Indicators of doctoral program quillty whatever the 
expi^essed purpoaes of the doctoral program might be. The third 
and fourth ali^ni indicators are obviously appropriate primarily 
for departments that place an emphasis on the preparation of 
scholars and researchers* 

ExMsples of Multidimensional Profiles of Quality for Doct oral ProgrMs 

As should be apparent from the overview of the many posFlble 
indicators of doctoral program quality exmlned In t iU study* 

{ 
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thart are no hard and fast rules about which specific quality 
Indicators should be used and which ignored* Instead, some indicators 
wara fotmd to hm dlf farentially appropriate depending on the discipline , 
the purposes of the evaluation > and so oii* Nevertheless , to give 
the reader a feeling for the deeper understanding that can be obtained 
about a doctoral program by examining many progrm characteristics, 
profiles of 23 indicators have been put tog'ether for each of the 
three discipllner in this study and are presented in Figures 11,1, 
31. 2 J and 11*3. For an explanation pf some of the "mechanics" of 
the prof ile~e.g, s how to Interpret percentlleSs how the score rnnges 
for each dlm^naion were calculated, etc*~sew the large^ enclosed box 
on page 11,20* Each figure presents profile information for two 
departa4antsi each pair was purposely chosen so as to be of approKiraately 
equal standing on two or mora major classification variables, such as 
size of the departmant or self -reported grogram emphasis on researoh. 
As th# reader can see, the department average on each of the character^ 
istlcs is plotted. Though such a procedure is useful for the purposes 
h/re— namely, the comparison of several different departments on a 
variety of different indicators of quality—we should like to emphr- tie 
that an evaluation of a particular department's stimding on any of 
th^se characteristics should b'^vond eKaminatlon of the department 
average and pay careful attention to the range of * ^lores" within a 
department « For examplas a dapartmeutal average of 10 public itlons 
per faculty membir over a three year period may mean that most 
individuals in the department ara publishing regularly or, conversely, 
that a few are publishing great deal and still others are not 
publishing at aU« 
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Interpreting thm Profiles 



The ladlMters of program quality are Identical in Figure 11, Ip 11.2, md 11.3* 
fhst iSi thm 23 ptQggM vsflablfes listed. ofT the left side of eaeh ffigure are^he aiaie; 
tot 





For each — 
dapataa'ata^ la each field is preseritfp . ^^^^ \fj^ ^ "^^^r^xJJ^^ 

*.*tha hliihest departMnt mmm from among the 24 or 25 departmenta in th^^ 
diael Una; 

*..tha dapartoent mean which is at the 75fh percentile of all department mems 
(that ia, the point below which 75 perceut of the other department mmmm would 

. * , thaHapa^rtnint mean which is at the median or mid-point (and below which half 

of fhm departments in our study would fall); 
.,.the departtent mean which Is at the 25th percentile (the point below which 

251 of the departments in that discipline in eur study would fall; and 
• ••the lowest department mean on the dimension. 

It is important to ^.mphasize that the nirobers represent department means or 
arlthMtlc averages. Thus* the distribution characteristics itt^ percentiles . the 
median, ate.) are baaad. In most cases, on either 24 department means (in chemistry 
and paydiology) or 25 department means (in history). EKceptions are, those data 
Obtained from alumni (entries 16-20 md 23) , where low response rates made it neces- 
sary to exclude some departments. For these dimensions, the data in the profile are 
' based on 17 departments Iji psychology^ 18 in history, and 20 in chemlitry, 

^ The meaning rhat can be attached to the nu^ers on the profiles varies from 
one dimension to the next. Some are self-explanatory, such as dljienslon #2 (average 
number of publications) or #5 (students' undergraduate grade average) or #18 (percent 
q£ aluMi with Jobs related to their graduate training), tost of them, however, * 
requliee the reader to refer to the queationnaire for absolute meanings, sines the 
dlmaialona ari*. often based on responses to one or TOre items from the question^ 
aalM. The specific location of each dimension is giVen in parentheses after each 
dimension description.' For example, dimension tl was measured by studenc scale * , 
dimmsion #2 by quastion 9-b in the faculty questionnaire, md so en. 

Where th^ dimension Is a "scale," that is, a score based on several items, the 
scores have been divided by the nun^er of Items in the scale so as to return the 
nw^ers to their original base. For example, the first dimension— student rated 
quality of teAchlng--l« based on student responses to seven different items in the 
questimmaire. (See Appendix F far details.) But the values for this dtaension 
on the profile have been adjusted so that they still correspond to the original 
response options. Thus, the highest department mean score on this dimension in 
ch«lstry*-3.29"e«i be compared to the response options which ranged from 4,00 
(for "encellmt*'} to 1 0 (for "poor"), and the reader can see that the highest 
departnent mtan of 3.29 on this dimension Is somwhere be^een a student rating 
of "gi d" «d "eiLcellent," By referring to the questionnaire in this way, the 
reader caa compare cchet dimension data to the original quest Iots. 



■ * 

figure 11.1 aompmtmB thm parformance of two large, elite, 
rMMrah-orlanttd ciiamlstry departments. In terms of the quality, 
of the faeultias at ^hese two Institutions, both are ranked quit© 
high (eofiparad with the other 23 chemistry departments In the study) 
with respect to their record of research (abllcatlon, research 
activity (e.g.j journal editorshipa, research grants received, etc.), 
and their reputation among chemists at other departments. But there 
is a very substantial difference betwein these^ two departmants In the 
student-fated quality of ^eachlng at the doctoral level » Department 

Is rated near the bottom on this indicator ^ \fhereas Department "T" 
receives a rating near the top* CThe mean rating of all doctoral 
students in Department "P" A between "fair" and "goodj" the mean 
rating of all doctoral studenia in Departitht "T" Is between "good" 
and "excellent," ^ v 

Similarly, Department "P" receives lower marks than Department "T 
on student -^rated faculty concern for students (Indicator #10^i the 

student^rated environmeiit for learning (Indicator #11), overall 

^ - ' V 

student satisfaction with the program (indicator #13) » and student 
ratings of the curriculum (IndicatDr , In geneial^ It is fair 

to say, that doctoral students are less satisfied with department "F" 
than Qepartment "T." 

The faculty ratings of De ^rtment "P" are more nixed*. They are 
considerably less satisfied with library holdings and slightly less 
satisfied with the other physical and financial resources, yet fchey 
rate the compatibility^ of the work environment much higher than the 
faculty In Departaent "T," and their' overall rating of the progMm's 
scholarly exctllttice (Indicator fUl) is abput the s-r . 



HoCG; In this ?xampir thy 
Ecji departments ar^ ■- =iEe iLirgi. 
place "extreme" €"""ihn -S on nrer 



Faculty 1 

1) Student-rat _d quality of -nr 

(? fcale ^n) 

2) Mean number of art id - ^ /book ■ IlU 

lagt three yeers (F=9-b) 

3) Peer^rated quality of nradu:'. 

(F-25-a) 

4) Research activity 

Students ' 

ST Undergraduate CPA (S=i3i 

6) Student consnitment/ni'^' " - 

Resourge s ' 

7) ^nculty ratings of lib*^nrv 'ifiMinv 

relevant to ;'n^ii liviC vl'^^ - = . 

8) Faculty ratings of 1 \horator jr f 

eQulpment needed for te.;iri i ^ lif', .in 
in your field (F-2=i) 
9^ Faculty ratings of over^^. i rid._tjii.i-. 
physical and finanrial rGsrnirct'-, 
doctoral pro^faUi L.t /^i\^v . ! I ■■ 

Environment : 



10) S tudent-^f ated f irfilry fM'5r>rf--rri > 

(S-scale H) 

11) Student--rated Environnient for .. irnln^, 

(S-scale ni) 

12) CoiBpatibilitv of fafzulcv wotk envirnnncnt 

(F-acals //2) 



Academic DCferingg * 
13) Overall student satis fieri on wU proj:r.H 
<S 



S-iicale ^3) 

14) Studisnt rating of eurrlculuni 

(S-scale #S) 

15) Student rating of assls tan t nh if. riL^r Lcnco 

(S-scale ^8) 
16 Aliaimi rating o*" disBertatinn p^^ptsrlen 'en 
(%^mr^^m in) 

Aluwii % 

17) Percent of alumni who declarod nr '.ram pre'p.;irpd 
them for career "tjctrer* ly well" (A=^^n, r 
IH i Hercenr wltn job hignlv r^-iHCfe^C la w ■■^\ii-iL---r 

training (A-24) 

19) Mean number TrEic '^/ha,ik rt^'Mew-. 
(A-27) 

20; Fresentaticr::i at rer'^on.il or natirmal 
meetings (A'"27^ti 

Sel ^^r4.tings of OvexajU Program _Qu a 1 1 ty : 

) Faculty self-ratingb of ovtjrall D^nfiraiTi 
scholarly excellence (F=scalt' i 

22) Student ratlngf I scholarly axe? It^nc^ 

of program (S- scale /^2) 

23) Alumni ratings of scholarly exceili*nccj 

of prosfiffl (A^scale #5) 
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Figure 11,1 pre^en^^ tiv ^laLa on t:.v^t^ pres ^ i/ i -ir^ J^parL..:ent 

icn are very Similar in v^Mt*. ?nai uppC'^-ra:\.;'0' repu ..^nionj , u ^ 

in which ".ier^ ip^^t^^^, to bt^ sor:^ inte:v:^Lin^ intprnil differences. 

Figure 11.2 compare ^- iwo a>iaaller, cunaidBraDiy less-pres t igiour 
apartments of '^isnory. indicated in .:he note at the top of 

Figure ^1.2, boch of these departments have about iO tull--t'lme 
doctoral students and p ' j "^"lout the same empp-sis on prepari..g, 
.^esearchtiiTS ..\d schoiars-/^ L^ t^rmure, they appear to be ^-^ery 



z; i 4 X d. L ^ i. L 1 1 L c ^ p cj 

act^vit^ 'indl^avcr ' -^d u"- h.:at:c ^tu-i'c^t ^nn^it as nn^pu^ed by 
^:nu^ ad'iatfe gc idtzS ^ ^ ^^id 1 c a tor -5). 

At this pint, however, the similarity berweein these twc 
departments ends, Depajt4:ient ''X" is more productive (indicator #2) 
and Its faculty perceives their students as being more ^onr^lcted 
and moCl^^ed (indicator #6). But the '*huma^ eness" of Depa* cment "X" 
is clearly nbt as favgrable as "hat for Department "0** as jvidfeaced 
by consistently low ratings by bOth students and faculty in these 
araas. Student ratings of the qu lity of instructiun at Departmanf 
"X" ara^ha lowest of the 25 history departments in the study 
(indicator #1) , and students' ratings of the facurty's crncarn for 
SLUdantSi the dapartmenf's environment for learningp their overall 
satisfaction with the program, t^ie curriculuT^, and thalr aspistant^- 
ship experiancea (indicators 10, 11, 13, 14, and 15, respectively) 
are similarly negative. In^^^Uon, faculty members in Dapartment "X' 
give -.omewhat lower ratings to the physical and financial resources 
of the dapartmant (indicators 7-9) , and considarably lov ^r ratings to 
the compatibility of the faculty work environment (indicaror #12), 
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Fa_c iixJCj; -. 

} Student = riited qualify :.■ t-n - = -. 
M Mfian number of af t [ zIq-^ ^^.'O^. 

3) Pger^raced qua] Itv or ^r/i iuU-/ : i 

(F-25-a) 

4) Research act vlty 

(F=scale 

Students ^ 

^) Urdergr^a aat GP. vj-— 

6 ) StudenC cotniTiirmi it/rnoi .v.it L in i h 

Resources i 

rfelevant LO their flt^Lci (F=i-f\i 
8) racuity ratingr of la: orit - r ^ 

equipnitint ntdLi^ J To: a*.^,;., l; 

research In your flotd (F=i= i ) 
9j Faculty rat trigs of nVi. ;icii>fiiuii 

physical and finaru iil '-^ho'itC''- 

doctoral progr in %cm:: ; 

r i £ unraen f= ^ 



10) Student- rated faculty couc^-v 

(S-scale ) 

11) Student^rat e-d p^-* f'^n-^^--" «^ * - ^- ■ 

(S-scale #1) 

12) Comprntibility of faciUy w rk 

(F-seale ill) 

Acac .^Ic Of fetings ' 
T f) Overall student sa t i :; I ac L 1 ni 
(S-scale 

14) Student rating of rMrrff mIm"^ 

CS-Scale #5) 
1^"* ^tudetJt rarlnp nf ,i ^ i ^ r nn t ^ n 1 n 

(S-scale i^^) 
Alumni rating of d issc rtrit Ion tsper i 
(A-Bcale i^l) 



17) Percent o' alumni who dGcI.ircd pr^^^; fan pri 

thes for eareer "extremely welL'" (A^4 n ^ 

18) P'reent Ith jof ighly related to nrod' ^ 

training (A 24) 
!N> Mean nunif or arr i r i e^ / r^nnK ri'vi*^ 
(A-27) 

20) PreRentat iof s ae /eglnn il f^*^ h^Uimm 

mtetlnR*? fA-27-f) 

e 1 f " r a t IngS^O f Qveral l Frogran f H^'^lj ^ 

21) . Faculty self^raf ^ngs of overal] r-r'nu"' 

scholarly excL .lence (F^scale 

22) Student ratings of scholarly extuli^ ^' 

of pfogrMm (S^Mcale ^^^2) ' 

23) Alumni ratings of snho-larly exec I U/nrt' 

of progi am (A--scale /^5) 
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1.66 

i 

i 

3.24 
I 

3I J2 



3.5/ 
I 

3.29 
I 

3.13 

! 

: .82 



0.95 
I 

0,86 

3.00 
I 

3I0O 
! 

I 

3.3] 

I 

3.25 
I 

3.72 
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productive y :Qro ^-rve thu in for : - d.i'^rtm u. . on 

both these indicators), I ::. s: ; , -v,: sat-^:i.lin- atu. 
iorale is dangerr'isiy low. 

..... ... -. „ ... ^ ^ v= -v^ f ,^ . i :j r f^- 

Q T ^ q r £^ j f V. . , rr; ^ ^ff ^ >^ H S ^ 

This exan:^:e serve?^ as an 
ex.:£lieni . . .^^ ir -i i ; .^i: "> " - ^ - j epar tmr^n t ' s reputation 

arr Deer-^ ^'^^ n-.-r.--^ > ^nf--- (-;rrGru .acM ab- deparLmert^^ 

pr. ..u.tlv y or rc-^-arc:: .K^-iv' •.;a^r^^Pnt "T' hns a Bcmtwhai 

in ^pit'- '"^'"^^ ^^^-^t ICS -.ill putii ^^"ion rate In th^ las: 

three -ears is the nighGHC of the 24 psvchnlogv departments in the 

study, and its research activitv 'evei i?^ ^ so highe: .nan LhaL . " 

Departr-nt K. (For a discussion of an alternative interpre Lat ions oi 

The fa.u.ty perceptions ot the repcurces avn : .able In Department 

faculty rarlnga rf the compatibility of the wo-k environment are 
siighLiy beiow Lilt; medid:., whr:.,is those Ot Uepartment 7. ar. abo>/t: 
Chfi 75th percentile. 

Student attitudes about u; e; - two rapartments are mixed, 
'stn mts in both departments ; -.te the ■.uality oi teaching fairly high 



lAHt thr^f vpAr- It-^- 
?er-f at ?tj y-: tir 



^ ' M •' j t ' "itT : ' T 

K>^- sou rces * 

7 ) Far- ^ ' rrif 1 nr- 

re Levant to tnt: 1 ■ 
Faculty ratlni:^; nf 
eQiiiprient nf^a it^d 
rr-H^arrh Ir. vnur 
9^ ' tc- lev r ^ tin>:H of 
pt{ 'Til and f 1:1 i 
doct al pTr^rarri 



10) S ttiden t^raf -ti *ic-il v 

(S-scale //4) 

11) Student ^ rat^d Fnvirrn^.^ ■ 

t^S^^ca L J 

12) Compa t i b 1 1 1 1 v o i f a c u 1 1 = 

(F-H ale ^^2^ 



■rk 



111 ^'f^raU student ^ati^f irtiim 'h pr.n r -- 
i ^cale ^1) 

14) Studfc t rating of currirnLurr 

15) Studt rating of ^Hr: is t.ui L^sii i p t'>:HrietUi 

16X Aluioni rating of H i pne ^ ' at I lui i-vpt' r ii nr*-'; 
^\-scale ^fl) 

them for career "extrfc'molv 1,, rl 

18; Percent with job ' . rrlattjd l<. i'radiMtt- 

training 

1 9 i Mean n umh e r of ■ r i r 1 / H ^ i » ^ r • ^ i « ^ w . 

20) Present at ions at regional or ^ini.. 
me^tingH (A^^^tj 

fk I f - r n 1 1 ng^ _o ^ CH^g r a 1 1 _ P r ^^ g rj!rrj ' \ ? 1 '! * ^ ^ • 
'1) Faci«?ty self ^ratings of overall pro>ir 

-rcl.olarly excgll^n ? (F-Mcale ^U) 
2J) StL^i'nl raLingB of scholarly exrolk'ni^' 

of program (S="calp ^^2) 
23) Alumni ratlwgii 01 ucholai ly i.'xt t^lb ..x %■ 

of program (A-scal*? ?/5) 
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low = .1 con-:ern for studentH Lut FCiJ^ "^v^ t^^t depn-t=n^nL a ^air'v 
high rating on environment for learniiv,, .p: sCudentH in 

Department E rare chelr departnw.t ; . opp'^site--high on raculty 
concern for stue^nt^, inw o,, ..nvi; niiLcnt t-r lenrninr (F-r det^iil^ 
on the ralationship between thes^ two qu^lUy indicatjrs, see 
ChAnter BO Thf: di rferent slz^^ of the tw programs is possibly 
relevant co thi-iv ii.donta' r-ting^^ the envirot.^^nt for learning. 
A nal inttjr^sriuf obaervat^or, aiout: the^^e two depa t t:!nef;ts 
:.o d:> with rhRi: yelf--a^inKB or p/og .r^ nuriity. botn la- 

HC nt.rartrnent I. th n bepartn.en E. In sp.te nf the fact that 

r^^:£-^.3rrb activity, 

^-.J :-T-d-i-'^ laidergraduate records were superior. We already 
mencUnad that rhe reputations of these two departments did not 
a^nccir jibe t^ thp r^urrtnt frets about the departments. It 
alao aeams that tl sel f ^r-i ^ . nR:3 of the f acuity and students at both 
dapartmants are ^mewhat icons: tent with soma of the inriicatois 
of quality presented here. 

Conglderatf on& B^iardinp th^^ - nf inf ormaLii^n ALuuL rro^rasi Quality 
lAfe focus of this study was on the feasibility of gathering mora 
drtailad information ab^u' doctoral nrograffl quality^ and ^.^veloping 
F thoughtful parspective on the meaning o( rhe naw information jnce 
.Htha ^. As a result, much attention has given to such issues 

as tne av.i. abdllty )f information^ rhe reliability and accuracy of 
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WTicit is not; adcrfiS^ed ■ i ^- ,y i-" fo^' : qu^sr "^ri:: 

abo t the u* liity of tn,_,re a^:^ ] ' :n^K . ' on ^^bout: doctorai 
pingrstp qn^lify t « (nf^r^^t^^;^ K: -js^'u^'i^ is it likeiv 

to more useful for ^0"\e es^aluation pui j.:use^. u*an others? By 
certain froup, lur^ rbm others,? 

It was nevn ■ jur intir^ioi; ■■ p-.o careJully these kinds 
of questionr. in ti-ln :;L'.iy. F:^sr a!i, it would hav^ been 
' uui^wii^L ^ifeU'fiLiML; auae. .^^tf- icuv. aucuc cnc^ utility or new 
In^az -ar bel'^/c :^ v^a.-. i b H i col^ticting LhAL Inf rmaElun 
had be^n th rough] / e..a:nip^/d. Tuf Important, Is claar 

that cor^. idc*? iens a- .iMt ^'^^ .^^f* c-f inf ^rTna^^nr^ r^gflrdlner nrncfr^Tn 
qt. Ht:\ are really m ^r^ phi: iphlLal questions than empirical ones. 
As st'^.. they are qutf^. tUr:t r:^3htfuny need to be diBCUSSt:d 

and de' da. le-^gL": by memberci or" the graduate conmiunlty who are 
concerned wit i qjallL/ in doctoiHl education. We can and have said 
much cnat ill lope/ully be r^c. pfui regjrding a variety of tecnnical 
Efipects of as3essit..| doctoral program quality, but the questions of 
"what ne it"*"" are ones which heed to be dealt with In another forum. 

Navarthelass p there ar- some perspectives about the utility 
of Inform. \tlo: bout ■ cloral program qualiiy that do emerge frGm 
the f. .uings of this st idy . They are presented here as a means of ^ 
sti '.la ting thougb-t^ nf p^rb^p^ _ en ^r;^ ting new Ideas Bhont dor t oral 
prog* jn evaluatirn. 



section In the discussion of the t.x^- pit cle;' ir 'T-^enia L =rnfiies. 
several referenceis were mad iz^ ^ngyi th.^i T^iy^:t rc:g^rded ...^ 

indicators of internal prLbl-ms or d^fiiculi in c^rt.iin department . 
As this suggests, potentially ii^f^fni Hi^tm^^ rf\-\ hp mad^ be'we^n 
internal inwicatorL of progra ^ quality and exterr 'I inuic .or^ 02 
program quality . 

External charac te. is l ics refer t:^ L..DSe features of a department 
or prngram I'lat tf?ud tu mor^; public and that also might bt 

chararreristt^— v ■ ; i , ther^-^:rv, ir.l'.J- ^ d-: Irti^^nt * s -'^c, 
rfeputatlonj phy^.cal ana financiaJ rt-sources, academic ability of 
the studenls* publication recoid-; ox its faculty, and the like. 
It turns out that these characteristics are also among the more 
traditional indlcat s of qualiL/ at the doctoral level, 

^.nternal -haractt 1 ist ics , on the other hand* refer t^ those 
features of a department that ar^ more private and less often 
considered ^hen one tninks about progra quality. These characteristics 
mijht include such indicators as student satisfactions and ratings 
oi the tef jhing and environment fur learning, faculty intarparsonal 
relations, alumni ratings of ^heir disseration experiences, and 
the like. On the example profiles presented in Figures 11.1, 
jA.Ss and 11*3, the facultVi student, resource, and alumr * 
indicators would generally be external indicators^ the environment 
nd academic offar.r'^ indicators wou3 1 tend to be InC'ernal ones. 



gen^z-liy the cas^ tbac thp extern al indicators o: pro^^rai- qualitv 
tend to be DOsitive:^.v courelated ^ itn eac:i othRr, a-a the internal 
indicators of prog an cua:^ tend t be correlated with each Jthe. , 
but the axtcrnai inUci^.r., ^ rulv . not cor elated with tho 

Internal indicators* Knowing something about the public character- 
istics of a departrrent_C it-^ reputation, for exampli.^ of ten perrnits 
a reasonably accurate ^utas about some other public characteristic 
(its size, its resources), duL usually tells you nothir. , about ity 
inner workings and more private quAl^ties. 

Diffe-rent uses and us^ Vh-j distinc- ion between inrernal 

and external dimeni^Ions of qu. lity is important to .= eep in mind, 
particularly as one considers the various uses rhat might be made 
ot intOLuiati^^ about do^^Loral pru^ am quality. Such information 
can be used in various ways, Jt might be used primarily for 
withln-program iniurmation, in which case it might be shared only 

ith ^hose inside thp dcpartmer^ or possibly within other areas of 
the institution but not wicn L^iose bkl^lii^I Lm Lue iii^tiLation, 
Or another possibility would be to provide the information to 
ralavant authorities outside the institutic.i to assist rnem in 
making judgments about e quality of the program. These two 
different models--lnternal use fo prograr. Improvement vs* external 
use in making judgments about p gram quality— of ^en call for dif- 
ferent informat un, for they obviously have dl. ferent purposes. 
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PArticuiar ly , che ■ ::r - - = : -.r- 

departir -t qj^lity, uS ne^..; ^.jj ^il^;-- . rudv b.. T:ort- raUvanL for 
prograiT; improyeiaenL Lhan ror ; i-^^^ntr -ado b\ a^^nc. C'lr^ide t.h- 
university. This very se::ti: ml in t-ct, w^. - expres^s^ a bv nanv 
progr:T:n facult^^ mc-nL-rs and r .ini.. -Hioi^ ylili ^^oi^' ^-jk during 
tne later i/tages . r.^^. They seldom doubted that informariun 

about such jharactGrisi a the environment an^ program offerings 

wa5 interestin>: and poiencially useful, but they fairly tinifDrmly 
saw as lata that ^^ou!': bt: shared outside th.^ univ^rt^ity onlv 
filter carerui cont iura .ii:i, 

- --iii tnwi..,c:L ,±,j u Lfiac might ^ b msde Lnturm^ti - i 

regard-ag tne vario':g ica^cr: of qualicy would to provid - 

the lugic being that such aformation would lead to an improved "fit" 
beLweon students c.nd de, artr .nts * One mech#nisr for such an 
Infomiatton flew woulr be to provide a guide to graduate pros^rams, 
stmila^ to the several Lhat are already available, but with mora 
detailed profile information ab- ut the depirtmentb ich as that given 
in Figures 11*1, 11.2, and 11.3. ..n^th s study we .d not give 
detailed attention to this possible use of indicators of program 
quail ty, but ve did ask a number or jople at the departments that 
participated in the study what they thought about this possible use 
of tht depar * T.ental descrl p^o^rs . There was a VGry mixed reaction to 
thn idea . SnTTie pointed out t naf ^nrr^f* nf the ^' f orma tion-^-par tlcularly 
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tnf kif.- we hPLVe : i. ^ i led "rxr-m.-i:" i s t r . ' i : i : y--wae^ 

airaady a Mdl-le ro t ■ \ o ^LuCents 1:^ i:.- iriouss gradu^.c 

chool gulden ^uc ^^IonaI;y : v .. iht: de;;i.. " t-tn l 3 [heiTisalves V^ve 
import^t was the :t i'^i -or;, .naiv^duai^ thr- ...cetna. 

indicaLors of d-^d -.^.a ■■■ -ilii-y ^i^.J.i " ^-^i i^cly the -ind of 
InfoimatiDn chat would hi ..-iue av- .able Lnose afepartkerts whc 
''lookfed g->od on s . :h Inui. .aors, ^at wo ;xd be withheld by those 
departLiiencs whose profi^-j^ -era icss r ^^^.itary. As a esult . 
they reai>^.uJ, student .:ncict43 jf departments w^. / 1 ^' p. ably lot b« 
improved by such a ^-actlue. Others argued, h^^wever , that dep ^rtmenLai 
coop' ation wo-jd a s.^riuus probit.... An any ^venl, tnia 

pO£4sr use nf pijgran. jrmiLion 1^ one that needs more thaugUL 

We have atten;ptec "o sunim r ^ ?ome of the different ways of 
thinking about tne ust ot doctoral /^rogram indicators in Table 1^.2, 
wh^r#^ fwd H'fterenr categorici or "fa^ilip^^ of quality indicators 
(extacnal and i- Cernal) are ey-mined t^i t respect to four possible 
ijse^ that might be mad-^ of the in^o. -tion, Conmien \ in this table 
should not I ^ Lntarpreted as racommendatiDns , but, instead, as otia 
way of organising and beginning to think about questirns no mint. 
Che use of prc^^ram descriptors in doctoral program evaluatio. of 



Concluding Con sent 

This st-dy egan with the positit r that a more rational approat:h 
to the asse? sme-^t of quality in do-torai ejucation was necessary if 
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taT'" judgT^e::*:s ri^ tr in^-^de ^-nut sini'^if'Lr u tor-^l pr eg rums and 
If pro^=.^am evaluation w-^ ? f?ver t-; b- me ,j vita- force in rraintaining 
and strengthening quality in doctbral educ?Lion in gereral. The 
ajib''" has fc cussed on uen^^lcal or mechanical questions — can better 
Informaticn about doctoral pi .-r^ni quality be obtained with reai.onable 
cosL and effort and, if s.^ , lat ioes it tell us? The basic 
messaga of the thousands of words and bers reported here is that 
much better information cau be obtaintd about program quality, 
and fehat it can be done '^ithou:: aat expenditureb ^ time and money. 
It does, howevt 5 require a ■:ood deal of par:xcipant interest ^nd 
cooperation, / ^ important k-'v mp.ovec program ■ evaluation ic: 

not only having moie detailed information about a larger lumber of 

regarding the status of other doctoral programs on the same 
characteristic-. In effect, the kiid of multiple-indicator program 
evaluation ini'del ^uggeFted her^ is no* practicable if only one or 
two dw^toral program^ ar - interested. The potential benefits of 
such an effort are la^rgely a fu: ction of the extent of widespread^ 
Interast within a discipline. Thu^ , it would appear that the 
future of doctoral urogram evaluation does not depend on the ^ evel 
of iiiterast s jwn by individual departments or p ograms in taking 
DErt in such ^ftoi , but on th? endorf e^^^^t , encoi.ragement , and 
direct act:;Dns taii.ii by larger groups, aLPOciations , or organizations 
withi grauuate edu^^auion. 
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This feCudv has resultiL^j i ..\ :i v;^;--- ^: i : ■ L :^ - u 1 li ilr.g^; 
some were unexpected, ^any were lu^t surprisir.^, other?i w-^^^ 
somewhat beruddllng. Ov. haL^n^-^, nowtzver , tne f^cts repcrted her^ 
provide an important rirsL step towarJ impru ed ductoral program 
^revaluation. The next step-^ are now up to t e craduare community. 
Whether the next steps are triken and, if so, in what direction will 
determine the nature of docl .^ral program evaluation practices 

K 

during the years ahead* 
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IN THE uxrriii) s: ATiis 



March 6, i9^5 

Dean Robert H. Bock 
Graduate School ^ 
University of \:iBconr n 
Madison i Wisconsin - 1706^ 

Dear Bobi ^( , ' 

As you know from our newsletter anC a report at the onnual meeting, the 
Council is QOnducting a study concerning dimensions of quality in doctora^^ 
, education under a ^ant from the National Science Foundatimu I am now 
' writing tp ask for yrur support-^and the support of several departments at 
your univt , rji ty-=in carrying out this project. Dot ails of the study and 
requests of tooperatlng institutions are summarized in the enclosed Fact 
Shet c. I think y^u will ^agr^ that the study is an import nt effort to 
broaden i.ie meanlirig and measurement of program quality, well worth the time 
it will require from someone on your staff and the members of three academic 
. departMnts * Let me tell you more about the help we need 

Ije Tv^ou 1 1 i ^ 0 - 0^ * T ^ , , ^ 2 T ''■'^ t ^ r C! c T 2, d u n ^ ^^z: '-^ 2 u ^ j ^ •- £ ^ ,^ *- i ^ ^ _ 
chemistryt hlatory, and paychology-^-^to participate in the project. In each of 
%h^P%^hr^^ depaitmentd we would then ask for questionnaire information to be 
-^prpyifid.by doctoral stud^ntn, gradu^ n facultyj and the department chairmarA, 
Ad^ically ^ the. question nal res will ask the respondsnts to tpell us about their 
dap-immeritp including such features as srudent^f acul ty relations, the extent 
to i4f£ph the program meets strident needs , and the like. The student and 
faculty questionnairas will take about i half^hour t? complete* The question-- . 
naires wguli be completed anonymously , placed in envelopes which we shall 
j provide , ar J returned to your office for forwarding t-"^ Educational Testing ' 
Sorvicfe , .^Jone og the Information will be identified with the specific 

regarded as srrictly r.onf ider :ial . 

The fmsedfkte product of this study will be a t^porj. tu ba ^^'^sued^ t6 the 
National beience foundation next ^al.lH^iCh copies alio ieing sgnt 'co %*ich 
cooperating inatltution and dapartment. The lor^g=range ouccom^, we hope, will 
be an incre 'SBd uaderstanding of the v^^ry comp4ex najture of the meaning, of 
qn'allty In graduate edun^tmrij 

how to go about assessing program quality. This may take 2he for., of workshops, 
manuals, and the like, aimed* a l chc use and interpretation cf assesstnent 
materials for program impr )vercrit ^ policy dicisiuns^ aad ^sxpanded inforrnbtion 
to prospective studyutB* 

If you or one of yuur assistants ecu Id serve as coordinato:^ for your 
institution, it would be an iiranense help to us. Thin person would be asked 
to make specifiy arrangements with r apartment ahr.irmen for questionnaire 
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distribution,, collect an z civ, it. 1'j.tcd r..,,i'. v i . : f rr.:.. Jy;..,rLr;;^:i: . 

mike inquiries aboi " for ^ Luat havv^ not Duen retuvieij proviuu u:3 w^t' .ht^ 
naiDes and adrtresses sk" ^ent gr'^ iuaLtis frar: tli^^se Lhree . ariments ar. 
;n general, serve sps the iainon Det '- ituj projucL ^; • cind your in?,i ..turion. 
Though it nay sOK;,d like a formidable .-i&k, we anticipate that the cntal time 
required f l r this activity would noi: evcueJ more rhan i 2 hours pe '^e' ^ for 
a period of about 5 weeks. 

* ■ . • 
^iay 1 suggest tne following series of sc-t.-.s as you conj-;ider this request-^ 

m , 

(1) . ^ad the enclosed materials, parcicu^-rly th^- project Fact Sheets to 
detarmine bow you personally feel aoout your in&t i tu . ion ' s interest 
in coopera^ i[i>; V 

(2) ^10. \e the chairTT^en of the three depar tmeii t a to explain tl^ nature of 
ouf request, and rher' send the enclosed materials (memorandum to 
^^partment chairir^an and ^"act Sheet) to the chairmen; 

( 3 ) as soon as ^ y o u b. a ve c he d e p a r t m.e n t ch a i rme n " s replies, omp le t e . n ' 
return the enclnsed fnr^ to us. Wp hope for iDnperafi'n from all 
three departmients at each uuiverbity in order to corLrol to fione , 
extent for differences in institutional environmer ^s - However, if 
^nly one or two of your departments can particip ^te, we would .like 
this information and will let you know as soor possible about 

Liie inclusion oi your in^LiLuticn in tiie studj^, 

t -■ 

(4) if you agree to cooperate in this imporcant project, the question^ 
naires and further instructions wili b^ sent to your cciordinator 
sometime in late March or early April. 

Siuce data collection must tar^ place this sprii,/' and we are av^are that 
the hXi^ of the academic ^r is rapidly ppproaching, I hope you can let us 
know your decision w ithin -fa few days. Please telephone collect to^me or 
Dr. Mary Jo Clark at Educational Testing Service (609) 921-90b0, Ext* 2795 
df you have questions or reservations* ^ 

• Obviously, we rtiink this project 1.^ extremely important. We a thaih^ 
you agree, and that you will be able to solicit the coopera' ion of the 
chainlstryj history, md psychology departments on your cflmp s. 

, ' Sincerely , 



J . Boyd Page , 
President 

^ Enclosurasi Fact Sheet 

itemoranda to Chairmen, Uepartments of Chemistry, 

History , and Psychology 
Reply Form and return envelope 
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On€ Dupont Cirrle, N.W , Washington. D C. 2nu * Phmi Artj CoJ^ jnj: 2j^-j791 ] mv^ PAGh 

m 

Mmorandum tot CHAIRMAN OF THE CRmThTRY DE^ART^ffiNT 

From: J. Boyd Page • Datei March 5, 1975 

This memorandum is coming to you from the deAn of your graduate school to 
invite your participation in ^ rGSMrch project sponsored by The CouiiD^l of 
Graduate Pchools. We hope you will be as enthusiastic about its prospects as 
we aie^^and that you and the membe*^^ of your department can provide * the .needed 
Information. 

istics of doctoral programs that- are reldted to the achlevraent of educational 
excellence. As part of this effort, we would like to obtain a profile of your 
doctoral progrM and opinions about the program from some of your faculty 
members, grmduate studei "s^ and recent plumi. These data would be'sumarized 
iOr your departaent and aualyzed along* with summaries from about 20 other 
doctoral^ programs In ch^istry asd a Wke niaiber of doctoral progrsms In 
history and psychology » Indlvidral responses would be seen only by meaibers 
of the research tmam and universities would not hfi identified In the final 
repdrt- The major purpose of the project is to develop procedures which coul^ 
bi used to Improve and expand the information available for doctoral progrm 
self-study and Improvement ^ progtam review and evaluation* and the guidance 
of prospective students « The accompanying Fact Sheet describes the project 
in greater detail « . ^ - ^ 

' ' / 

In addition to consul tatis4^ with tt American ChMlcal Soclaty, the 
project's development ard the design of data coilection procedures have 
been guided by an advisory conmnltte^ of^ rhOTlAt^ composed of Glenn B^rchfold. 
Maasachusatts Institute of Technology; Charles Lester, ^ory University; 
Wlllitth McEwen, University of Massachusetts; and Cheves Wallingp University , i 
of Utah. ' ^ 

Tifflfe d^andj of the project Include i.-out a half-^hour for faculty 
mCTfcers and students to complete their respective quest iDnhalrast arrange- 
ments in youi office to distribute and collect questioanalres and follow 
up on non-rasppndents (respondents will put their names ^ the outside of 
sealed envelopes to assure confidentiality), and staff time to provide some 
basic descriptive information about your doctoral program^s* Quest ionualres 
will be available for distribution In lat ^ March or early 'April, 

Please look over the accompariylng Fact Sheet and then let your graduate 
dean know If your dep»jtment will tie able to participate In this Important 
efforit. We lo^^forward to working with you. - 
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Concerning a Studv ui Zov..ti. 



Purpose of the Study 

Tha major purpose of th . r— iccc is to gain a better understanding of the 
meaning of qualify in ^cadu^ze education an! suggest some ways the assessment 
of quality might be impiuved. It goes beyond judgments of quality per se to 
id-^ntify infcrmtion thar would be meauingful ^rom prugram to program and 
usaful for program self-studv and improvement , internal and external dacislon- 
making, and the guidance of p D^pective students. Doctoral programs in three 
dlsciplinas will be studied on the awumption that dimensions of educational 
excellence vary somewhat from cne area of know%-.dge to anOLhar. Se-eral major 
program goals (for instance, preparing researchers, teachers, or practltion^ 
will also be considered. 



A receut natioTlFil survev of c^raduate school deans jdentified a variety of 
progratf characteristics they' felt were essential far judgrnents about educ 
tional quality and some acceptable ways to measure these ^laracterist ics . ^ 
The deans* Jud^ents have been used to construct questionnaires to collecf 
information about individual departments from faculty members, enrolled 
studants, recent alunmi, and departmental records. About 20 programs^ xn 
each of tha , fields of chemistry, history, and psychology are being invited 
to cooperate in a field study of these assessment procadurM. Data collection 
from cooperating programs will take pldce in March and April, 1973, 

^ata collected from these varied sources will be su^arized by department 
md by field. Cooperating dapartmants will rec^iv« profiles of their rcjulte, 
but all responses from individuals will be confidential and no department or 
university will be identified by name In reports of the study. The su^rized 
data will also be analyzed for relationships between selected cha-^acterlstlcs 
of doctoral programs and dif farent sources of information, In order to specify 
more el early the range of factors that should be considered whan maklnf ^ 
Judgments about .quality . Data collection InstrOTents will be revised, based 
on the study results, and made available for use by departments and univer- 
sities undertaking self-studies or program reviews. 



Sponsor 



The study is jointly sponsored by the Council of Graduate Schools in the 
Utvlted States (CGS) and the Graduate Record Examinations (GRE) ^ard. It 



^ *A full report of this survey has been distributed to manbers of the 
Council of Gradimte Schools; additional copies may be obtained by requesting 
GRE Board Research Report No, 72-7aR ^Tha Assessment of Quality in Ph.D. 
Progrms^ A Preliminary Report on Judgments by Graduate Deans) the , 

Graduate Racord laminations Board, Educational Testing Service, rlnceton, 
Not Jersey 08540, 



1.9 funded 'yv grant of 31^5,D'jO ir^^^i t 
Council. Thfj --r'-iy if beini; -^onnu^.^c b': 
thp gpne*^."!! "lire ^n\. of a sre^rln^- ro^=:^' 
Board.* In aauil th^re an c^d^i^^u-; 

of the disciplines selecurtd r field ^ 

Un i V ersit 1 e s_I_n v i t e d to F ; i f lei n r q 



Ahout aO universities that award doc to rat ep in chemit^L ry , hlstor" , ari-l 
-^sychoiogy are being invited tc participate ... this field at;id . We hope 
i:hat all three departments in each university wili agree to cooperate. 
Since universities were selected randomly from a srrat'fied list oi al] 
eligible institutions* there are no preconceived notions about the profl'° 
that should emerge for any given department, T' g list includes universities 
broadly distributed geographically , somfc large ind some small programs in 
each field, aid some new or relatively unknown programs an well as some with 
top national reputations. 

Da '- fl Collection 

The key to this project is reliable information ab. it each doctural program 
from a v.^riety of sources-- faculty meroberSj enrolled doctoral sti ientSj 
recent alumnij and departmental records. The following brief questionnaires 
will be used to collect most of this information. 

F a cu 1 1 y Quest 1 o nna i r e : Distributed on campus to all department faculty 
members who teach doctoral ^^t .'cn'Sj this questionnaire asks professors 
to char/ terize their ueparcment by responding to a number of statmerits 
abou^peopla, programs, and policies and to report their current ^rofas-- 
sional and research activities* Faculty raembers also are asked to rate 
appropriate programs in other universities, ^ha form can be completad 
In about a half -hour. ^ 

fft ud ant Q ues t lonna 1 re i Distributed on campus to all experienced doctoral 
students (usually in their third year or beyond) , this quastionnalre 
provides an opportunity for enrolled students to characterize the 
deparcmant along some of the swe dlmanslrns provided tor taculty and 
to evaluate the program in relation to their needs and Interests, A 
fevj questions on student backgrounds and objectives also are included. 
The "form can be completed in about a half-hour. 

Aliaml Questionnaire s Vary similar to the student questionnaire but with 
additional itms on emplo^ent and other post= degree experiences, this 
questionnaire will be mailed by the Council of Graduate Schools to a 
sample of those who received dDctoral degrees from the department in 
1970 to 1972. Campus coordinators will be asked to obtain appropriate 
n^es and addresses from departments or alumni offices. 



*The steering confalttee is composed of J* Boyd Page (President of the 
Council of Graduate Schools and Project Director)* Philip 1. Kubzansky 
(Boston Unlver.: ..cy) I Charles T, Lester (Emory University), Sterling HcMurrln 
(Univarglty of Utah)* Ralph E* Mor jw (Washington University)* Lincoln E. 
Hoses (Stanford University), Michael J* Pelczar (University of Maryland), 
and Herbert Welalnger (State University of New jrk at Stony Brook)* 
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Dfepartmfental P -oflle: Departm&u c' ^ir=-:.^n r rnair 1 pr ^ 4cn t i ve^ will 
asked tn provide some basic J?-: ^riptivi. i r ' ; : rna t iun abiic each (ifiparr = 
meTit--number of students arH ■ -v A^y r^f^^h^-s,. ^rii^ j 1 r.nq d^rn, fl-auC:^] 
asslstanct_ daca, spec i.al i sac iunri ^ ut;^rn^t^^ grf^r.cea, f^i^cernuni ^. i ^ iduaLt^i 
and so for*, h. 

Respon sibilities of Cooperating Programs 

The graduate dean at each university \^ a ked to appoini one perse/, to serve 
as local coordinator for the proiec^ Appropriate data coUeetio'! macer.als 
will be supplied to and returned by the coordinator; he oj she will also be 
briefed as thoroughly as possible to handle questiuns or protlems that may 
come up on any given campus * 

The coordinator will make arrangemeats with t e chalrmari of each department to 
infora students and faculty about the project, distribute questioraaires 
collect completed questionnaires (in sealed envelopes in order to assure the 
confidentiality of responses) , ana follow up on non^respondents * Suggested 
guidelines for these activities will be supplied with the questionnaires, 
Departme: ts will al^o be a. '<ed to complete trie uepar rmentat rtuLil^ lullu. 



Benefits to Cooperating Prggr^arn^ 

Each cooperating department will receive a surLmary report of quest lotinaire 
responses together with a e iramary profile of all cooperating departments in 
the ^ame tielcl, it is :iopeQ tru cais in^ urnt^L i urt wIlI ^ t/f iii^erafet to 
department members concerned with self-study or program evaluation; suggea- 
tlons for Improving the utility of the feedback will be invited* A m«ber 
of the research team wilL visit most cmpuses some timfe during the project 
to talk with department members about the research and Its findings. In 
additionp we hope it may be possible to set up some workshops on uses and 
interpretation of the data after completion of the current project. 

Next Steps 

4 

not the dapartments of chemistry, history s and psylihology at hl» or her 
university will be able to cooperate in this effort to collect data on sane 
quail ty^-ralm ted characteristics of doctoral programs. We also nmmd am 
estimate of the number of faculty members who teach doctoral students and 
the nuaber of graduate studerfs who are In their third year and b^ond In 
each departmrat, so that we can supply an appropriate nimber of question^ 
naires. Questionnaires naist be distributed and returned this spring | 
therefore^ it is important to have your response within a fe%^ days. 

If you have questions or would like to discuss some ^pect of the study» 
please write or call collect to Dr. Mary Jo Clarke Rehfljarch Psychologist , ^ 
Ed'catlonal Testing Service^ Princeton^ New Jersey 08540, Telephone 
(6Lp) 921-9000, Ext* 2795. 

March 5, 1975 

AS 



f 



EDIICAT'ION AI. Ti^BTING SERVICR 



CAaLE^EDt CTESTSyC . 



I*RI NC feTOlS. N, J. 08540 



HBmorsaidum to Campus Coordinators 



About*. COS Dimensions of ^ality 
Project 



Datei April 4, 1975 

Fromi, Frojeat Esssarchers 

Mary Ja Cluk 
Laoaard ^Ird 
Hodnay Hartnatt 



We ware ve^y pleased to learn that your imivaralty muld be par^lclpatitig 
in the Council Graduate Sehools pilot study mf^ proctdui^as to d^serlba 
quallty^ralated charactarlstica 6i doetoral programs and that^^ySu wtlU ba 
coordinating data collection on your eWpus» 

As you will sea from the enclosed mater|.als, we have sent a brief 
inemorandum to each cooperating departmint chairman on your' campus ^ together 
with soma guidalinas fbr the distribution of the faculty ^nd student quas^ 
tlonnalres and sample copies of these questionnaires. « Wa urge you' to read 
tnosa matarials fairly closely ao that you can axLSwn questions and determine 
the best way to coordinate data collection on your campus. 

Boxes of ques^fennaires are being sent to you by th^ ETS shipping depart^ 
ment. They will send you a copy of the shipping order ^ which will Indlcattg 
the way In which the materials were sent md their eKpectad arrival date* 
A Xerox of your fox:m^ Indicating the jiumber of qUastlomalres you raipiestad,^ 
is 'Enclosed with this memorandum. Note that m^ii deparnieAt should receive 
^hn 7^t mnrm questionnaires than requested to cover spellaga and to^ use 
when following up with nonrespondents« Also^ note tluit each department should 
recalve a supply of return envelopes" that Is equal' to tha ntmbar of question^ 
nalres they receive. We cau Supply more questionnaires at My time If you 
find that you will need mo ire » • 

Department chairmen should return completed questioraalras In their 
sealed envelopes 'to you^ so that you can return them to us by RIA ^press 
Collect in their ;origlnal shipping boxes. Hater lals for this will be Includad 
n the shipment ^o you*, Wa would like t^i> have the completed queBtlonnalres - 

returned to us by the end of May If at all pOBslblk. 

As Indicated In the Guidelines, there are several things we would like 

for you to do for uSt " 
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NmDFandtim to Cimpus Coordinatoifs -2 



1: Dssl with questions or problCTia concerning questionnaire distribution 
md' rat urn. Probably you can answer most questions without further 
^ infoniation from us. But don't hesitate %9 phone collect , espeelslly 
If a question comes up in more than one department . Our goal is to 
bbtalA a faculty and student view of each depattment that Is as 
comprehensive md complete as possiblei this Is more liiportMt than 
'whether part-time people are included or excluded, or th0 exact 
ntaber of r^inaers that may be needed to get a questionnaire returned, 

2. Encourage the departments to keep track of, who receives quektion- 
nairesp the procedures they use to follow up on nonreipondentSs and 
my particular problms encountered in data collection. .We do not 
need names of ^respondents, buV one of the purposes of the field study 
is^o assess the feasibility of data collection. Therefore, we are 
very definitely intbrested in any general problems > resistances, 
ausplclonsi or roadblocks that are encountered in ^efforts to obtain 
Information directly from ^culty members and students. 

3. ^¥e' would like a university catalog and coplSt of > brochures or other 

materials that describe t^e aoctoral prof tam^' in chOTistry, history ^ 
and pgychology. The Guidelines ask department chairmen to send you 
copies of descriptive materials , for you to forward on fb us; if 
theie-^o not arrive within a few days, you might want to theck with , 
department offices about them, 

4. We /need i list of doctorates awarded by eAch department from 1970 to 
1972. Thpsi lists will be, uped to conduct a mail survey of graduates 

' who received their ^grees three to five years ago. You s»ay have such 
lists available in youi^ffice, or you My need to get thm from the 
departments or from the alumni office. , We plan to mall questionnaires 
directly to alimil in order to reduce the project -s tlTO dCTands on 
cooperating universities*. However* If you or your department chalmen 
would 'prefer to contact t^ie alunml llrectly, or if you must contact , 
th% aluriflii before releasing their names and addresses, we would hp 
glad to provide you with an appropriate niflnber of questionnaires and 
postage-paid return envelopes. 

Please let us know which procedure you prefer. If you plm to supply 
lists, we would like* them as soon as they are available. We will 
send you pmple copies of the alimni questionnaire as soon as they 
^ are printed* ^ 

5. Departmental profile foraa will be sent. to you for dlstrlButlon to 
the department chalmen in about two weeks*, Unlike the studint and 
faculty questionnaires, these forms do not have to be completed 
before some of the students and faculty leave cMipus for the aumer* 
however, we hope that department chaiirmen can complete th^ by the 

• end oi May or shortly thereafter* You may find that their task am 
be shortened somewhat by providing some of the information from 
records or reports in the. graduate dem's office* 

# ■ 
O ' ■ ' ■ ■ 
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MrniQTmn&am to Csm^w Coordlaators -3-- 

6« s Oei# fiddlClmal group of potmclal program reporters, not pravlously 
■tttloud .In Mterlali about tin project but of eonsld^rable Interest 
CO uSf Is diseratlnued students. We raeognlEe thst dropouts are 
pwtleuJlarly hard tm idntlfy at> the doetoral level ud that it nay 

* ba avm aora difflQid.t to laoate thaA the, other h^nd, 'they uy 
provlda a view of a doctoral study program that la qulte# different 
tt&m that rvf othar partlc£put8» ai^ therefor^ i| may "be worth soma 
* apaalal aCfoM to obtain their opinions « ^Thoraforai wa poaa the 
folloving quaatlona to you* Could you or your' department chairmen 
proylda laa vlth the names ^nd addresses of parsons ^o "dropped our" 
of the fhsD* programs In ehmlstryi l^lstoryi a&d psyctelogy over 
' .the iaac eoupia of y^tml About hoif many people i^lii this bei by 
dapartmnt? Do yoiu have evl^dencai pro^ or con, about (a) the 
faaa^blllty pft-obtali&l^ questionnaire raturfis''ffom tAase paopl#» 
and (b) the likelihood that tfielr responses i^uld differ from those^ 
of anroliad doctoral sti^^ints? * , 

Though fhla Hat of tasks may appear nther formldabiei wa anticipate that 
data collection on^most campusea vlll go smoothly apd that coordlnailon will ^ 
no^ raqMlra tiore thm a few hours of your tine. Heedless to 8ay» we appreciate 
your help and" Intarast in the' project. We stand ready to help In any way^we can, 
Ineludins a canpus visit If this would be useful. Agaln^ don't hesitate to 
phone collect vlth any quastlohs or problraa. We look forward to working wlth^ 
you. * . 



Enclosures: 



Xerox of. Institutional Information 
, Menomdum to departnent chalmen 
Suldailnes for oft^campus data collection 
Faculty quest lonnalire 
Student quest lohnalre 



PM I ^jc errors*, fs. J. ohr4o 



(lABLE^EIH'CTE&rSVL 



* 



M^orandum tc Chalraeni 
DepaTtments of* Chmlstk^ 
History, and Psychology 




About I COS Dlaenslons of Quality 
P^ojaet 



Date: 



Datei 



April 4, 1975 



From I 



Fro J sot Researah^rs 



Mary Jo Clark 
Leonard Balrd 
Rodney Hartnett 



We are very pleased that your department will be partlclpatl'ig In this 
Council of Graduate Schools ileld study of doctoral program characteristics 
associated with educational eKcellence. rhough most of our contacta In the 
next few weeks wfll be with your campus coordinator i we want you to know that 
we appreciate your efforts on behalf of the study. Please feel free to gat 
Axi t ch with one of us If you have questions that cannot be answered by your 
coordinator. , 

Three itams for yeur Information are enclosed with this memorandOTii 
' , it 

1, Guidelines for on-cmpus data colleetlon, * specifying-ths faculty 

^mrabera and students who should receive quest lomialriis and outlining 
some suggested procediires for distributing asid collecting the ques^ 
tionnalreSf Some other requests for infprmf.tion from department 
chairmen are also reviewed In «the Guidelineiii 

2^ a sample copy of the faculty questionnaire (gteen) | 

3. a sraple copy of the, student questionnaire (blue)* 

Supplies of the two questionnaires are being sent to your coordinator^ 
who irtU distribute them to each departments In the n«t few days you may 
want to establish procedures and prepare a cover memorandtm so that the ques- 
tloimalrM can be distributed as soon as possible a£ter th^ arrive* However p 
as you will see in the Guidelines , t!.^ tljae of distribution la up to you. 



The (Suidelines contain some other requests and comets that you will 
want to review* If you still have questions, please contact your coordlnat 
or one of ua* , We hope the data collection goes fmoothly and that you have ^ 
an except loaaily high ratt of returned questlouAalres! 
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IMilNCKTON, N. J. 0S540 



CAMPUS ^ COOHDINATO]^ 
DEPA&TMmiT CHAIMffiN 



About I ^id^I^^s for On-Cmpus Da^> 
Collection — Projeet on 
DlflmalonB of Quality in 
Doclior&l Education 



Datss April 4* 1975 

From I hary Jo Cluk 

Resmarsh Diractor 



Tn thtsm quldelinas wt suggest some proeedures that might be used on a 
eampus or in a department to collect the information that la needed for this 
study. Not all of the suggestions will be appropriate for every situation i 
but we hope they will clearly convey our data collection objectives oo that 
you can devise appropriate altarnate procedurefi when these do not fit. 



^eq^yeats of DeMrtjrent Ch^iirmen 
^ fc ^ 

A food return of coo^leted questionnaires is »trmely Important for the 
reaearch goals of the study mad for the usefulness of rfsponse sumiiaries that 
will be return^ to you. We, expect that the beat response rates will occur ' 
when a departmei^t chairsan supports the* study ^ Informs his or her faculty and' 
doctoral students about it^ and encourages their participation. ^Therefore» 
we urge you to take whatever steps will convey your support and encouragement 
when the quest 4^|||&alr as are distributed. Some^ possible mechanisms might be a 
CDver mMoran&ym accompanying each questldnnalrep a brief statemmt about the 
study^at a meptlng of factilty memberi and/or students » and a rtiftlnder at a 
department colloquium or other gathering that may occur shortly after the 
quest ionpalres are dlstVlbutad. You might want to point out how the study ^ 
or feed^ar.k^to the department » eould contribute to department development or 
^program' reviews within the university . ; If there are concerns about conf IdM- 
tlallty of responses s you might want to reiterate that all quest loraalres 
will be retumad to the researchers in aaaled envelopes and that no indlvidu^ 
responses will be read on campus. 



The n^t nost taportsnt' Inf luMce on respfense rates will be the procediirea 
adopted for questionnaire distribution and retUm. Probably each department 
has a list of faculty m@&bera and a list of graduate atudents whlciti can be nised 
to form a master list of the persons who should receive questionnaires. 
Distribution might be by campus mail or some other local syst^ that gets 
the right piece of paper to the right person iflth 'reasonable efficiency. 
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(U»S» nail aaa ba uaad'i of ^CQuraey but che projact has no funds Co aovpr 
this aost*) Sona ^antral l^aation should ba daslgnated ^qt ^uastionnMira 
raturn . (sueh as tha dapaitment chairman 'S^ off lea or a box in a student or , , 
faculty lomga) I a^ somaona will naad^to check off raturnad quaationAalras 
OB the distribution lists. We tops you will sand at least one rmlndar to, / ^ 
thoaa irtio have not returned quastlonnalres about two weeks after they are ^ 
dlatrlbutadt Mo^^e about this below. 

• . # 

Tiiiltig 

The complated faeulty and studant quest lomalres ahould ba ratumad to ul 
by the and of .Hay. Within this constraint » you may aatabllsh whatever dlatrlbu- 
tlon and ratum, schedvle seras bast for you. Csapus coordinators smA dapa^rtment 
chjilrman may want to dacide on a ci^mon schedule » ow dapartmmts^ may dparata 
Indaffandantly. Conflated quastlonnalres In saalad envalopes should ba ratamed 
to your campus coordinator, who wil^ retuiti thra to ETS in bulk shiptaiants. 
Plaasa let your c^npus coordinator know as soon as possible if It appaars mat 
you r*ay need to go beyond the end of May in order to obtain a good rate cf 
res'i^onse from either students or faculty, 

I . . ^ ■■ ■ 

Fafeu3|ty Quastlonnalres (g^reen) 



One faculty questionnaire and one return ravelope should ba alatrlbuted 
to each faculty' member in your department who teaches doctoral ptudants. 
You will want to specify a place and deadline for their return in your cover 
memormdtm. Our goal is to bbtfin a faculty profile about the doctoral program 
that is as complete as possible. Therefore, faculty In each*. sufa^sp^clallEatira 
ahouldi be locluded. Faculty members ^*lth' Joint appointments, part^tlma appoint* 
^nmts,lwd adjunct appointments should be Included if they are raaaonably well 
acqualiited with the departmmt's oparatlon and program and the^ contribute 
substantially to your doctoral training effor|s. Thry may ba omitted If ybu 
feel that their Involvement with the department Is relatively minimal. 

■- ■ . ' * 

Stud^t \ quaatlonnaires (blue) ' 

One doctoral student quest ioim^ire and one return ravalopa shotild be 
dlatrlbuted to each established doctoral studrat in your dkpartmmt^ Usually 
• thMa atudfnts will be In their third year or beyond of graduate study , 
althdugb Sana aacond^ye^r students mlght^ be^ appropriate In soma departmtf ts. 
In gim^u^ the questionnaire should not be dlptrlbutad to firstly a A^j|£Mdrat8 
tq^ ^udnta work^^ toward terminal maater^s'degreesi ^ayond this, we will 
ace#|i(^youif detamlnatldii of those who can give the moat complata nd accuiata 
raai^^v^ to quaatlons about the character and quality of your doctoral . ^ 
pr^lnMiw. Hiaafe Include studmts In all of the. sub-spaclal^^-tlona within 
the mptf^Bant p md lAclUde part-time as well as full-time studants if there * 
la an appraciahla' part^tl^ enrollment^ Try to Include soma atudents who 
are wrltiAg dissertations and some who have not yet raachad this stage# If 



Epp^o^riati and aVAilablai Ineluda ntudentc who ara ^playad full' tlmt while • 
ttieF MflqilMe thslr' diss totat Ion ^« 

IkiliiiiAti thm help of yojie o| the leadars your graduate student group 
flk^ ba a uatt^ My ^aeoitf ags^ a high rata of student rssponae. A co^mt 
fMMtaadiSB mMtt^aglng participation and giving Inatruetlons about ,ra turning 
tlM ^untimm^tm Bight b« fros ^ha drt»artmsnt chalnemt a' studant leadari 
0r b^thr k aMdnt- daparcmantiy. ars'l8<;aut might da aakad to nonltor qu^^^l^u- 
oaln rati^u c^ganiitr a follow up to nonrespondaats » or. In other ways 
to anMuragti participation In the prujact by other student i. ^ ^ ^ 

foiiow Pp . ^ . ^ 

J', 

' ' Va hopa |fpu. vllx ba able to follow wp on both faculty' and student non- 
raspondttta at laast oncsi beginning |boat tvo weeks aft rr the questionnaires ^ 
are dlatrlbut^* This might be done by marorandumi Eaquirles 50^ group heads , 
phon^ calls » rmlndera In seminars or classes, or other procedures* You may 
iwant to end a second copy of the quest lo^£)^lre to those who do not reply by 
a given "datai «tra copies have been sent to you for this pu'rpose» 

Confidentiality ^ * , 

Hota ^hat envaldpi has ?aan .provided for each qCBs'tlonnalre. Cc^pleted 
^eatl9nnid.res should be sealed In these mvelopes before they are returned to 
you, nd thcf sealed envelopes re tamed" by you via your coordlnato:f to U8# 
lasi^ildants are askad to place their nraes on the outslda of the envelopes 
only ao that you can Identify ^ose who naa4 to b# minded to complete ud 
,r^u?n their forms | no name Is quested on the questionnaire. After data col- 
lecMon la conplatedi we would to know the nuiAer of faculty members and H 

studanta to whom questionnaires were actually distributed , th# folloi^^p 
procaduraa that were used, the ntofter of quastionnalres retumadp and any 
bias in the returns that Is apparent* to you (e^gvi oost members of a particular 
sub^s^eclailzatlon declinad to participate or a poor response #rom part-tiise 
paople). Wa will not request a list of pfcrsofis who did and did not complata 
a questionnaire • 

•i - ■ = 

Papartmental Profile . 

The fo^^that '^lll be used to collect^ some basic infomatton about each 
departvint will be distributed to cmipus caordlAators In about two weeks. 
Most of the Infoniatlon requested on the fox^ should already ba ^^allable In 
the dapf rtaant or daan*s of flee | guidelines will prdvlded to simplify 
complatlon of the form ai much as possible « In the Mantlma» It would be 
h\lpful If each d^partmenii^^hat^im would send the coordinator one copy of 

broehurai report , or piece of recruiting literature that deacrlbes doctoral 
progr^Da In the department » outlines dagrae requlrments, reports^ on jobs of 
graduataat or In some oth^r way conveys information about the department that 
would be relevant to thl* study. 
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Aluwnl Quest i^malre ^ . ' , 

the enpua coordlA^itof has hem asked to craplle a list of ^persoiis 

who rsealTed dootoiral dagfeas from your dapartm^t from 1970 to 1972. Thtsa 
lists will hm used to aoaduet a mall survey of doator^l progrm gr aduAtes 
thCM to fivs ^ars Bl%mt receiving their degrees « In some eases y dei^rtttemts 
may prsf ex tn aall ques^iotpialres thraselveSt vl^h a request for partlelpatlon 
that is mors personal than a form le^L&r from thu Couacil of Gradi^l^ Schools* 
If you would like to contaet your altuml directly, please 1st your qporddfinov 
l^ow theVuQbsr of qasstldnnaires you will rimmd md we will bo 'glad to supply 
thear» - 



If Tou Have Quesiiions ' . 

_ 11 _ - . 

The empua doordlaator for ^ the project will be able to mm?er many of 
the questions that are likely to come up in relation to dat^ aollectlon In 
individual departments. Hwever, ff you have a problTO that cjumot a Mswered 
locally » or if you would like to talk with one of the researchers directly * ^ 
please telephone collect no Mary Jo Clark or Leonard Balrdt Educational Testing 
Servlcei, Princeton, New Jersey (609) 921-9000| ext. 2/95 or 2792* 
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I i*Ui Hit ( iFslSit 



To: CampM CQOtditmtQtB^ MB Project on 
DlMOsloas of Quality In Doatoriil 
E^oatloii 

9 

Subject: 1* Departttental Profile 
. Other matters ^ 



Date I May 15^ 1975 ^ ^ 

Fromi llary Jo Clark 

laseareh Dlreetor 



Inclosed are four copies of the Departmental Protlle fom Md n 

aceoi^anying m^Borudw tor department ehalrmen In eliCTlstry, history » and 

fsyeholo^t One copy is far your Inforaatlon; pleaiie dlstrlhttte the others : 

to your cooperating department chairmen with a request that the form pm 

return^ to you aj soon as It li complet^. 

^ TP ^ 

As you Imow.. we had hoped to get theie materials to you a good bit 
earlier In the St jlng^ but various probl^s conspired to hold th^ up. 
An unplanned benefit may be that departmental staff will now have more .ttoe 
to aosplle the requested data after the iiressures of spring cMmene^ent 
have abated. Obviously, however , we no longer mpeet to have the completed 
f orsis retiirned to us by the eni of May, In the memorandini to the cha^^n, 
I teve suggested that they complete the form by the middle of June if at 
ail poaslble. ' ^ 

It ^uld be helpful if ym could look over the 4orm before sradlng 
copies to the chairmen to identify any questions that could be answered mosf 
efficiently from iifomation la the graduate dep's office, or an instltu^tonal 
research or* registrar's office, so that you can suggest these resources to the ° 
ahairien. We would also like for someone In the deai ' s office to revifw each 
cd^leted form t^iefly before returning it to us, to be sure that the figures 
given by the department are qonplst^nt with your knowledge of each proitam. 

As noted In the chairmen's memorandum, we will have one TOre brief ques- 
tlomaire for each chAiman to^ complete--a few questions about some specific 
aspects of the progrm-s offerings md prouedures, and some Judpients about 
the effect of recent occurrences on pMgram quality* We hope that these ^ 
really will reach you within anothM^ek or so. 

Now on to other mtters. Several of you have stnt us lists of 1970-1972 
alwl and descriptive materials about each progrM; th«ik you for your prompt 
re^oonse to these requests. If you have «ot yet sent us these materials , we 
hope'thtft you are in the process of collating thOT and that they will be sent 
along shortly. 
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Tot Canpus Caairdlnators , CGS Project -2- 



Sottii of you hava raqumsted supplies of the aldmnl questionnaire so th^t 
ope or npre o2 your deparcmenta cm pend th^ dire€t?,y to their graduates « 
It Aow appears that these will not be ready to maH out to you until about 
the^ seoond'waek in June # The questionnaire will be eight pig?,s, the same 
' size as tht student questionnaire i and will by pre^ foldeu tc fit In a legal- 
si^a emralopeii In ease you would like zo start the }^oq£hu ui addraf^sing 
envelopas^ ^ 

V ' 

Wa hope that the discribucion and cullection of the student aod faoulty « 
qu^atlotuialres is going along smoothly J If' you have not ehecked reoexitly 
ulth dapartmants about r^lnder notices sr^ other follow-up proaedures with ^ 
noQ^^espondents^^ you might do this now« ^ 

It has e^me to our attention that some of the questionnaire shipmenta 
dl^Ji not have materials to use when returning the boxes « Therefore p we are 
epc losing a set i^f these materials (mailing labels and gunned tape) with each 
o£ these mraiorandums. It would also be helpful if ^you;^ could mark each box 
for the attmtion of Ms* Theresa Jones, B-016, The boxes should be returned 
%y liEA Express Collect* Ypu need not wait until you have'all o{ the question^' 
nair(li ^before returning a boK| when one^illa upi send it along. Qiftestionnafres 
received after you have returned the bulk of thra should be sent by repilaf 
mall to me* ^ 

Agalns don*t hesitate to phone if you have queitlons-^609^921»9000p 
tot. 2795* We have beeir very impressed with the efforts the crapus coordinators 
haVe Ade with regard to this pt^oject that have come to our attention. 'We 
main confident that the project will be worth the effort* ^ 
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To: Chai£iMn, Dmpmttmntm of Uhmmlstry, yatei M^y 15, 1975 

History t and Fsyahology 

From I Itery Jo Clark 



SMJsct: DepartMntal Prof 11a for 

GGS Dlmaasiong of q|uallty Projact 



Rasaaroh Dlreccor 



you know from our previous eommunleations about the Dimensions of 
Quality la Doetoral IducatlQn projpet, wa nmmd some factual IfiforBat ion about 
your departmant ud doctoral program so that these charaetaristlos can ba 
conaidared In ralatlon io Inforffiatlon bbta^asd from your departaant's studantSp . 
f acuity mOTbars^ and^racant *altoiA The aceompmylni DepartTOatal Proflla 
rorm has baan designad to eollaet standard information with a minimum of special 
^affort on the part of departraental staff m^ers. This memorandum atteapts to 
anticipata aone of tha ^uestlon^ that may coma up ragardtng the form, mi 
provides mom of 6ur reasons for askljig for theaa pa^rticular departmantal 
statistics. 

F^cuity staffing . This section asks for the number of faculty mambars in 
the departMnt^ t*ha nunibar who work with doctoral rtudants, and a full-tlma 
. aquivalant figure which is hare defined as ttie total ninabar of full-time 
faculty plus the fuU-tlma equlvalenrnof part-time faculty i Faculty mambers 
who are aasipied full time to the/department should be conslde^ad full time, 
whathar or not they spend all of this time In teaching and whether or not 
* part of their salaries are covered by outside funds. Joint, part-time, .and 
adjunct appointments should be considered part time. 

^ Questions e through 1 ask for lnform«ion about only those faculty 
.ffi^ers lAiO tesch or supr'/isa doctoral students* One definition' of these 
faculty m^iibers might be tha faculty members authoriied to direct dissertations* 
Do not ittcluda faculty who teach only undergraduata or terminal aster's 
. courses even thoq^ they occasionaliy attract a dactoral student registrant. 
Our purpose hare Is to obtain a coimt of the ntOTber of persons vrtio ara available 
to work With doctoraf students md some information about them as a group. We 
^ utlclpata that thlp information is most easily obtained from departmantal 

records, althmigh you may want to turn to the dem-s office or an institutional 
research office for soma of it. a 

Faculty raseardi activity . Iphe items In this section reflect progrra 
' characteristics"* that graduat^ dems thought were Important to taiow about when 
^ considering tfce academe quality of a program. We are frankly iicertain / 
^ whether dapa^^enta cm answer questions 3 and 4, concetming the tofal n^ar , 

' ' < , ■ ' ' * . ' ■ 
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of publleacions ud research proposals which h#.va been subaitced and tuxided 
over the last three years* without a great deal of elerlcal effort. Slease 
* at least uswer. question 4e (actual dollars available to the department each 
y^r or r^earah). If you cannot answer ^e other ^questions with reasonabls 
accural and ^qpmdlture of effort, a cometit on the last page about the 
jirobl^u thay pose for you would be helgjul, 

PepMtttntal slge * This sect ion asks for susaiary poults of faculty mCTb<er8i , 
liistructionai suppDrt staff m^ber^s and students at different levels over| 
the past three years « AgaiUi full-tine equivalent is defined as fuil^timel 
plus the full'tlM equival^t of part7time, persons in the program. The 5a 
f Iguife f6r Fali 1974 should be the same km' the FTE figure in la. Our purpose 
here is to get some no^on of doctoral student enrolim^t in relation to 
other aspects of the departmental program. ' 

This section also asks for degrees awarded by the departmrat at each 
level over the last three years » and for some dollar\ figures t^ represent the 
f in^cial resources available to the depertment for Its total program and for , 
its doctoral progrra# The check-list of it^ra* included in tlia education and 
general ejqiense budget figure will be used only to get a rough estimate of 
the comparability of budget figures supplied by different departmeilts In 

' diff ernt ^universities. Information about dollar mounts available to the 
department for financial aidp assistantshlpSp rad research are requested 

p e\S€i^et^ and therefore should net be included In the figures reported in 
tKis section. , . ' 

Enrollments and^egr^es by area of specialization * This section asks for more 
deljbaii^ infortiatltm, about the current number of faculty members who ^ teach 
doctoral jfCudents by itmm of specialisation | the nimber of enrolled doctoral 
students In each area; the number of degrees granted in each area ill 1973-74 1 
and t^e approximate length of tlAe to the degree in each 'area. The total 
ni^ber of faculty mmbers should equal the figure reported in le# We hope 
^his information will allow us to do some .sub^analyses for each disclf'^lne 
aceording to major area of specialization | in addition » the faculty f 8%|ident 
figures wlil help us d icamine the extent to which faculty and student ques- 
tlotm^re rteapondents are representative of area specializations within each. 



AdmlstioM data . These questions ask about the nuiober of applicants for . 
ataisslont the number admitted, the ntmber rarolledp and the charmcteristlcs of 
the nterlng class for each of the last three years* Mote that we would like 
this information for your doctoral progrm onlyp rather than for all graduate 
students » Both st^ents with baccalaureate degrees and thos^ ^o alre^y hold 
master-s degrees should be included in a through h| question lOi asks for the ^ 
nifliber of ifiawly e^olled doctoral students who had already completed master *s 
level vork«^ 

Ftoancial support fdr students . This sectira asks for the ninAer of doctoral 
st^^ts who s^m receiving finucial assist^ce this yearp and the total dollar 
anbtffits of fuidds from various sources for student support. 
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Bttployrcilt * This saetloQ asks for the Information you mt^y have about the 
positions obtalsed by your ddotoral degree reelpients in the last «two years. 
We wtmld also like your estimate of the number of these first Jobs that were 
direetly r@lafced to each graduate's aaad^lp field of studyt 

GoMmtS a The last page Is for your coments concerolag the statlstlos you 
hKwm piovldttd* Tou nay be uncertain whether some of your figures cu be 
fairly ooap^rad with figures prb^ded to the ss^e question by programs at 
other ual^ersltle&i If sOp we would like to taov hsfw you arrived at your 
flguree afid your reservations about their comparability t You nay need to 
iMVe soM ItMs blank because the Information Is not available or would take 
n Inordinate nount of tiMm to compile | we would like to Imow about probl^s 
you may hftve with these Items. In some cases » you may prefer to «upply copies 
of internal reports that provide essmtlally the msmm Information rather thau 
nmmv the questions as they are posM on the form. You nay id^sh to Indloate 
irtileh pleeee of Informatlbn were obtained frra the graduate dean's of flee or 
aoae othw office at the imlverslty rather than from* departmental reMrds. 
Anything that would help us Interpret the data and get the iMst complete and 
accurate picture of your doctoral progrra would be helpful . 

-St^Mrller comunlcatlonp we asked for copies of statements » brochures ^ 
Ot filers that describe your doctoral program and degree requlrtmentSs If you 
have not already sent these to you^ crapes coordinator to be forwarded to us, 
we hope that you can do thlg now. { \ 

We realize that this Is a particularly busy time for departmental offices' 
and that our Initial request for the Departmental Profile Information by the 
^id of Hay is now unrealistic. However ^ we hope that you will be able to 
complete the form fey the middle of June« Please rAura it to your campus 
coordinator In the graduate dean's office^ who will send It on to us« 

Thanks again for your efforts In the distribution and collection of 
student and faculty questionnaires as well as the compilation of Information 
for this form» tee more short questionnaires to collect some personal Judg- 
ments from department chairmen about their programs p will be sent to you In 
a few days and will complete our data collection efforts In your department # 

Your campus coordinator may be able to help with other questions that 
come up as you or mmbers of your staff complete the Departmental Profile 
form. If you are still uncertain about an Item, please telephone collect 
to 609-921-9000 for Mary Jo Clark at Extension 2795 or Leonard Bated at 
^tension 2792. 
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Toi Callus Coordinators, COS Projact on Datei May 27, 1975 

Dimensions of Quality In Doctoral 

Education t 

Subjacti Information from Department Chalmen Fromi Mary Jo illkttf 

^ Research qireetor 



< 



EnelQfled are t e last fomis foi data collection from your chOTistrys history » 
and psychology departments^a request to department chairmen for some additional 
Information about their programs and their judpaents about program quality in a £^ 
specif ic areas. Please send one copy to each of your cooperating chairmen. You 
\ will note that wa are asking for chairmen to return these fotyis directly to us, 
* rather than to you, to protect the confidentiality of their responses on some itma* 
We hope that the forms can be completed and returned to us by about the middle of 
June. 

Because of questloni raised by some ol che coordinators, a few comments about 
the financial Information requested on the Departmental Profile form may be helpful. 
Most persons wbuld agree^ I think, that this ln£orn»tlon would be nice to haya as 
one set ot variables in any etudy of program quality. But we are also ^are thpt 
doiiar figures can be computed In various waya, can be misleading, and are politi- 
cally sensitive* In particular, it may be difficult or impossible to separate the 
coats of doctoral programs frrom other departMntal costs* We included a few 
financial items on the form in the hope that the information would be available^, 
aad alao as one way to find out whether or not this is so. But these itmmB are 
obvloualy only a small part of the total study; department chairmen should feel 
free to omit them, or add caveates, if they question the reliability or compara- 
bility of their data. . 

tet me be Mre specific about the financial ItOTS in question. Probably the 
most troublesome item is 5(j) on page 3 which asks for the approximate dollar ooimt 
of the department's education and general expense budget*that Is used to support the 
departtentU doctoral program. Departments may be reluctant to make such aa^ etotlaate 
without an agreed-upon formula for its calculation, and the detail that would.be 
required for a precise calculation is beyond the swpe of this itx^f. We wouad like 
the information if the chalraan can make an estimate that Bmmm reasonably reliable | 
if sycfti m estimate cannot be made, the item should be left blmk, 

' ' ' / = 

Item 5(k), dollars for doctoral programs from sources outside the imlv^rslty, 
may be more available but may also be difficult to eeparate from funds use4 to 
support other departiflental programs. One possibility here ifould be to provide the 



^ total dollars available to the department from outside sources (not including 
financial aid and- research) with a note that it is not possible z6 separate ou_ 
the aaount of th^ funds that are used for the doctoral program* - 

\ 

It^B^ll(c) and (d) on page 6 ask for total doiiars from the uni'*"-"''ty for 
assiatan^pips and other financial aid to doctoral students. We a _.ame tha th^a 
figures '©111 reflect the dollar amounts of financial assistance actually awarded to 
doctoral students this year, regardl of how tlie funds were allocated, and there^ 
for© should not present any particula. problems. However, if ^here are uncertainties 



^ It"^ 5(e) 4 total dollars from nnn university* sourceF for sCudent aid, 4 s al 
the sum of ards to doctoral students this year. 



so 



Items 4(e) (page 2) and 5(i) (page 3) ask for votal dollars to the department 
for research anjj for ec^ cation and general expenses. Since these items do not 
request a breakdown tor doctoral programa uuly, tuey ahuuld be relatively easy to - 
answer. However, again, if ther is reluctance to share thi.: information for use 
in the research , the chairman should include a note to this effect* 

I 

I hope that these comments may help you respond to questions about thest: .terns 
that cdma up on your c^pus, and may allay any concern about their Intent or 
purpose. The entire study is exploratory^ in the sense that we are attempting to 
find out how useful the information is in gaining a better understanding of the 
neanlng of qua^lity in graduate education, but also In the sense that we want to 
find out how accessible this information is and how easily it can be obtainad. We 
eertainly don't want the Departmental Profile to be d tiresome burden on deparrment 
chalrBen or meniiers of their staffs. If the Infomatlon requested cannot be put 
togather with a reasonable expenditure of staff time and effort, we would appreciate 
thalE telling us that, along with their comments on the difficulties they encounterecf * 

If you think the department chairmen would be Interested in these additional 
conments about the Departmental Profile form, I hope you will sand them a copy of 
this memorandiai. 



Thanks again 'for your help and support, 
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Memorandum for: CGS project campus coordinators Date: June 13, 1975 
at Colorado, Florida Scate, 

Michigan, Princeton, and from: Mary Jo ClarK 
!T-i^^„,,c'^ RpBR,irch Director 

Re: Alufflnl qnestionnalrp« to be ,iiled out from 
your universit;. 

Accoopanylng this memorandum ara the number of ^""^ 
return envelopes requested by onu or mT depart;;.;,:.^;; : ==?er = tiTiO " 

this project at your university. Perhaps it would be helpful if I repeati 1 
here a few guidelines for their use: 

1. The tariet pgmilation is persons who received doctoral degrees in 
1970 1971 It 1972. Ouestionnair"^ should be sent to-alj. of the 
oeopie who 'received degrees in these years 1 ' - fields you are 
surveying (chemistry, history, and/ot-pSychoiugy; with tne following 
axceptlons^ 

a,^Names with foreign addresses should be omitted; 

50n?^ly vlflhl*^ addressas ahould be omit tad; 



c If there are still more than 100 graduates in these years from 
' any one department, sample from the list to send out 100 question- 
naires. If the list is by date of degree and there are 150 names 
with useable addresses In the U.S., it would oe appropriate to 
cross off every third name and send to the rest. You may need to 
develop other sampling procedures appropriate for your situation. 
The main , thln^ is to include graduates from each of the three 
years and various aub-speciallzatlonB within the department. 
2 Envelopes are provided for alumni to return questionnaires to ETS at 
project expense. We expect to do at least one follow-up on the • 
• questionnaires we are mailing out directly to alinml. If r^u plan 
to do a. follow-up. we will need a list of the names of persons to 
whom you sent queatldnnaires . Me will check off returns on this list 
and send you a progress report about three weeks after the queatloti- 
nalres wetis ^^aL out* 

3. A nice touch, if you can mant ,e it. would be a cover note frc. the 
, ^ 1 „ ^ii.mril fn romrtlete and return the 

questloanalre. 

4 It would be helpfui if you could let me know the dateCs) when your 

aliBMl queBtlonnalres are maij.ed and the nimber jrou send to graduates 
' of sac^ department. , n 

Thanks again for your help, and don't hesitate to phone tf you have any 
questions. ^ ^ A24 " ' ^ . ■ . 



Dr. Allen G. Marr 
Dean of the Graduate Division 
University of Calirorni^ at 
UaviSs Caii^ornia y^bib 

Dear Dr. Marr; 

T ist spring the t'outic i I or f;racluatp Schools in the United States enlisted 
voui cooperation and tliat of vour doc tor. il prog rams in ciiemistryj History , and 

lance or quality* The cooperation and patience of your campus cQordinator arid 
those assoGiated with eacii of theBe departments on your campus has Haen except- 
tionally good. ^bviousiy, th Ktudy ould not hr^^^e taken place without their 
ganarous assistance * without your support* 

We can now begin to share some preliminary study results with you in the 
form of reports on questionnaire responses by advanced graduate students, 
faculty members i anH recent alumni in eacli of the cooperating departments* 
Three copies of each report for departments at your unlv'^rsity will ba mailed 
to you within the next few days. We suggest that you send two copies of each 
report te th- apprsprintc dcp-rtn^cnt :hairn:an director cf graduate ctudlcc 
and keep one for your own use. 

As you may recall, there are two major purpc es of the study. Tha first 
Lh to imptuve uui uiiu* i-aLaadiug ui Cim iitfe^iniiig uf v^uallLy in graduate eduu^tluri 
by looking broa^ 'y at program purposes , environments, and practicas as well as 
reputational :^ntings. Analyses to ^shed light on thasa questions are now under- 
wav; a summa? / of the final proiect reoort will be sent to vou in the spring, 
A second^ more ''applied" purpose, is to develop ways of describing doctoral 
programs that will be useful -^or progrtam review and improvement. Obviously, 
the faad>-ack reports to cooperating departments are a pa. t of this aspect of 
the project. Your commontr concerning nctunl potcntinj uses of the informi^^ 
tion, and any other observntionB or suggestions u erning tlie project, ould 
be most welcome. 

If, after receivinp, Liu* f outline i-portSi yon liavo any quest Ions ah Oiit your 
resulrs or tlie project in ^uMieral, i - "iope you wl I L telcii .ione or write to us. 

Sincerely ^ 

»^Mary Jo Clark 
Research. Psychologist 

* 

cc* Suoan Jcrgcnscn, Campus Cc-rdina^cr 




f)r fhf ' Nf^fi'l fh^ i^'r- 



Dean CbArles T, Lester 
Graduate School 
Frrinrv ITnlversitv 
Atlanta, Georgia 30322 

Dear Dean Lestar: 



Last spring the Council Graduate HchonlB u\ tlui United States onlisCed 
your cooperatipn and that your doctoral programs in ch-rai^^try, history, and 

excellenre or qualitv. Th- cooporation and pntience of your campus coordlnacor 
and those associateu with each t ■ these departments on your campus has been 
exceptionaily good. ObviouHly, Li,« study cpnl \ not have taken pi -a -.-i rhm.t 
their generous assistance, f^r without your support. 

We can now begin to shnre anm<= pr^l mi ,ary -^tudv results v i you in the form 
of reports on q- ostionnalre responses by advanced graduate .tudents, faculty 
members, aud recent alumni in each the cooper at ing departments. Three 
conies of each reporL F»r departments at your university will be mailed to 
you w*ithln the next few days. We suggest that you send two^copies ot each^^^ 
report to the appropriate aeparcmenL ul .iiii.idii ui. director of graduate stu.ie? 
and ^OTg one . for your own use . 

As you may recall, there are two major purposes of the study. The first is , 
to improve our understanding of the meaning oi: quality in siiiuu«L« aJu^ation 
by looking broadly at program purposes, environments, and practices as well as 
reputatlonal ratings. Analyses to shed light on these questions are now under- 
way a 6.unur.ary jf the final proioct r-porc will be sent to you In the spring. 
A second more "applied" purpase. is to develop ways of describing doctoral 
programs that will be useful for program review and improvement. Obviously, 
the feedback reports to cooperating departments are a part of this asp'^ct of 
the project. 

We hope the reports will be of interest to administrators, faculty riGmbers, 
and students concerned with the opt ration and development of these r ograms. 
To help us evaluate the potential usefulness of materiala such as these, and 
to accu- itely Interpret these particular questionnaire results, we would like 
to visit your campus for a day or so sometime in November or early December to 
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Lonnr 2^, 1975 



talk with you and department reprt3sentat ives abouc these matijrs, • will 
phone you or your campus coordirraCa£_an the r xt two weeks to discusE specific 
arrangaments for such a vis^ * 

If, afctr receiving the feedbci reports, vou have any quGHtions about vour 
results or the project in general^ we liope you>v/ill telephone or write to us. 

Sincerely, . 



QCi Marion Combs 



. iary Jo Clark 
Kose'rch pHychalogist 
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QUE TIONS FOR CA>ffUS CONFERE:NCEb 
DI!^SIONS QUALITY IN GRADUAVE LIjUCA. .^)N 



Any questions about the reports you received? 
^ it&ffls of particular interest? 

Likely utes of the reports in your departmi nt -= 
faeulty ccmittee^? 
student eomittees'r 
deeis ion-making? 
progrm improveTnent? 
what would make them more useful? 

useful in providing Info ,to prospe'itlve studanta 



What other experlenc^:i have yo had with pr'JfeTam rs^ ew? 
State involvement? 

processes ~ 

evaluation — 

how migh^ they have been more useful? 

In your opinfon, what would be the best way to assess the 
quality of a doctoral program? 
reputational ratings? 
visiting coTmni ttees? 
student/ faculty/ alumni reports? 
success of graduates? 

have such surveys been done? 
what's criterion of success? 

Other pr^^dures you would recoimendT 

What about program purposes in relation to the 
assessment of quality? 



If there were regional workshops to discuss issues 
coneernad with the assessment of ) doctoral programs, 
parliaps based in part on the results of the eurrent 
rasearch» what issue should be discussed? 

who should take part in these discussions? 

(get copies of raportSi if possible.) 

Have you done any validity studies on the use of 
GHl scons in admissions? 



Notai also try to clear up questions about data supplied 
QtL the departmental profile (inconsistencies » 
' missing data, etc.) — and^ if no profile has ^een 
mmat top try to get basic Info rs nmher ol €aeulty» 
nraUM&t at dlffermt l^els. degMes grants p #tc 



4 



Appendix B 

Selectxon of Institutions to Participace 
in the Study 
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Chemistry, History, and Psychology 

Unlversltief ara arranged in order of total 'number of doctoiates granted in a 
recant threa-^yair pariod^ from lareest to smallest* Each group of universities 
accounts for one quarter ^ the totai number of doctorates granted by these 
Institutions in these years. Randomly selected universities invited to par-* 
* ticipata ^n tha rasaarca are marked with an asterickp 

[ Three^Year 

^otal Ho. Total Graduate 

Group I Doctoratfcd School Enrollment 



University of Illinois 2779 

♦University of Wisconsin, Madiscyi 2617 9281 

^University of Michigan 2285 9460 

Michigan State University 2130 ^252 

Ohio State University 2105 8298 

♦University of Minnesota 1759 7446 

♦University of California, Los Angeles 1752 7714 

♦Stanford University 1643 4954 

♦Naw York UniverBity 1600 16239 

Group II 

Purdue University 1558 6997 

♦Colimbla University 1480 3000 

Harvard Unlvar?5lty 1442 2652 

Coifnell University 1433 3343 

, Unlverslty^ Tfi^aSs Austin , 1323 6031 

Peimsylvania Stafe University 1289 6411 

UnivMsity of Southern California 1279 4836 

unlvar3i,ty ui gk«iii j^^u^ •^^^.'^ 

University of Chicago 1249 ^ 5623 

♦University of Missouri , 1239 4377 

♦Univeraity 6f Pe^ns^'lvanla - 1168 4006 

University of Washington 1146 6^7 7 

.♦HDrtherriwMtern University 1123 2532 

♦Unlvat-slty of Maryland 1098 9065 

♦University of Iowa 1055 4857 

Group III . 

♦Indiana University ■ 1040 4565 

Yala University 1018 2316 

bnl^arsity of Pittsburgh ' 1003 9348 

♦Florida State University 962 3465 

UniVarsity of Oregon 943 3025 

University of OklahOTa / 930 " 5566 

'Case Wastara Sasarve UnlvLrsity 876 1968 

Univarsity of Florida ^ 870 3746 

Univaralty of Horth Carolina 839 44j3 

^Univarsity ©f Colorado - 811 ' 3723 

Syraeusa Urtlvarsity 809 3779 

University of tamassaa 786 5594 



Group III (continued) 

lutgeri University 
*Unlverfllty vof Kansas 
♦Princeton University 

Stnr? St Buffalo 
- University of •Georgia 

Catholic University of ^erica 

Duke University , 

University of tol^ona 
*OiashoBa State University 

Unlversltv of HH«sachn^ettR ^ mherst 

Jolms Hopkins University 

University of Nebraska ^ LincolTi 

Group IV 

Boston University 
Waviifc State University 
*Loulslana Stata UniversiLy 
University of Virginia 
University of Rochester 
Southern llllnoia Univaralty 
Washington University 
University of Connecticut 
Arizona State University 
♦University of California, Davis 
Umlverslty of Alabama 
Broim University 
UttlVfrstty of Northern Colorado 
" Tople University 

Smiat L«mis University 
Washington State University 
University of Hew Mexico 
Umlverslty of Cincinnati 
Umiverslty of Notre Dame 
University of Kentucky 
Vaoderbilt University 
Tulane University 
Carnegie-Mellon University 
University of California* San Diego 
♦West Virginia University 
Brandeis University 
City University of New York 
University of Califomla, Riverside 
University of Wyoming 
Lehigh University 
Brlgt^a towig University 
University of Houston - 
Ttfas Tech Unlve^ity 
Kusas State University 
FordlM Oniverslty 



; .^ree-i iaivi 
Total No. 
Doctorates 

7P - 
769 
756 

749 
746 

74r 

679 
671 

6 '^'3 
654 
m I 
643 



Total Graduate 
Schcal E nrollment 

9297 
3951 
1421 
5278 
3991 
3995 
1898 
5612 
2941 
5379 

3151 



633 




4053 


627 




7873 






T n C C 


62 / 




J ^ 


616 




3470 


608 




1626 


595 




2663 


527 




2720 


510 




4353 


494 




6117 


479 




2630 


472 




2283 


469 




1237 


465 




1597 


442 




6815 


429 




2720 


428 


■ft 


1640 


42^' 




2951 


413 




4301 


412 




1219 


411 




2764 


385 




1088 


374 




1030 


360 




i218 


335 




1199 


333 




.2775 


321 






319 




2723 


310 




1198 


299 




1287 


294 




2030 


290 




2307 


^ ai 




3301 


269 




2680 


261 




1090^ 


250 




1753 
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Total No, Tot^i^ Gr^dusL 

Doctoratefc School Enrcllment 



Groyp ly (cantlnued) 

University wf Miami 238 2298 

George Washington Upiversity 228 1362 

University of South Carolina 225 4320 

University of Dalaware , 220 ' 2201 

Univarsity of Dewer ,216 766 

Unlverplty of Hawaii 211 3567 

*fiaory University \ 210 9 51 

University of Mississippi 203 1142 

University of Callfornlai Santa Barbara 202 

University of North _L.nkota 198 1175 

SUNY^at^AlbMy*^ ^ " * 188 ^ 429f 

Kent State University 182 ^ 3113 

StJNY at Stony Brook 167 4355 

^Boston College 157 2135 

Texas Christian Univarsity 152 1070 

University of California* Irvinr 139 930 

. Tufts University 

Northern IIUl-Is University 112 17333 

Iryn Mawr College 106 465 

University 6f Montana - 90. 695 

University of California, Santa Cruz 87 307 

University of Akron 84 . ^ 2038 

University of Maine . 77 1575 

Clark University 72 382 

♦University of Toledo 67 1712 

University of Nevada ^ 61 1099 

University of Arkansas 60 167 

University of Illinois, Chicago Circle 46. 1542 

StniY at Blnghamton ^ 40 2056 



^Most of thm Information for this tab^e is from th. .graduate Programs 
a nd AdmlaaionB Manual 1971-74 , pu: lished by tha Graduate Record ExMiinations 
Board tha Council of Graduate Schools in the United States in 1973. Jhis 
velifl&s raports total nimber of doctorates warded In 1969-1972, and total 
gradusta school enrollment in fall, 1972. All informatior for eight institu- 
tions that are not Included in this edit ion of the Manual are from the 1974-^75 
edition and represent doctorates awarded 1970-73 and total graduate school 
anrollment fall 1973 (Brown, Ohio State, Texas Christian, U. of Arkansas at 
Fayattevilles U* of California at ^anta Cruz, U. of Illinois at Chicago pircle, 
U, of Hipsouri at Coluabia, U* cf Virginia).^ Three additional Institutions 
are not in either editiTO of tht^ Manual (Brari^eis, Illinois at Urbana, of 
California at Santa BarbaraJ; for these institutions, total numoer or doctorates 
awarded a ^ from A merican Universities and Colleges , 11th ed. (Ame^can Council 
on Education, 1973) for the fuur^year period 1967^1970* 

Universities that award doctorates in one or more of the three fields 
cf interest were eliminated from consideration for one of several reasons: 

1, the ijniversity reported fewer than a total of 10 doctoratas per year 
In a recant three-^year period (usually 1970-73); 

2, information to the 1973=74 and/or 1974-75 Manual indicated maste'^s 
or specialist as the highest degree awarded In ona or more of the 
fields I ^ 

one or more departments not listed in either edition of the Hantial 
and no doctorates awarded by tha university in this field 1961-1970* 
according to toerlcan Universities and Colleges , 11th ad. (most ©f the 

category) I 

no doctorates awarded in one or more of the flelda 1961--70, accoi'dlng 
to the lith edition -of ^erican UniveraXtl^a and CQlla£#fi a and no 
doetoratea awardad in tha sMe field (s) 1970-71, according to Earnad 
Degreas Conf erred 1970-71 (NCES, Dept* of Health Education and' Walfara, 
19/3), even though the field Indicates "doctorata" as the highest 
degree awarded in the 1974-75 Manual . ^ 

Tables on ^'Earned Doctorates 1961-1970, by Subject and Institution" n 
the 11th edition of American Universities and Colleges and the National Centar 
for Educational Statistics report on Earnad Dagraea Conferred 1970-71 were 
usad to idantify universities that award doctotatas in the three fields but 
ara not listed in either the 1973-74 or 1974-75 editions of the G raduate 
Programs and Admissions Manual . 
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Dt^an ' s KaLirigb -jL i\ j^rani wiidrdc Ler l^L icrf ^ 
buldes to Questionnaire Const rut: l ion 
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IFffORTMJT PROGRAM . lARACTERISTICS AND ACCEPTABLE hEASURES 
AS RATED BY CRAJDUATl DE^^^S 



In the suniaer of 1973, a pmal of about 60 graduate deans rated 
the ImpoTtBtic^ of a great mai y Ph*D, program characteristics for 
JudgMnts about educational quality and then rated the adequacy or 
dccaptsbllity of several alcernaLive ways to measur*^ each important 
characCSTlstlc» ^ ^e ' program characteriatics aqd maa ires that ware 
'"^ndarsad by the deans are sumarlzed in the following t^Mas accord=- 
isg to^ f our potantlal sources of information i university or depart* 
asQtal racDrds* faculty questionnairas , Btudent questlonnairas ^ and 

these tabulations (as opposad to 43 availabla for tabulations raported 

in the project repurt) and rasponsai from deans at high^prestiga 

uni\ iitles are tabulated separately for contrast with total responses* 

R^dijigr Chi Tab les 

lua tables star^ with Indicators impj^rtant prograni critirr . :eris tics 
that should be available from university or department records. The nLfffl-- 
ber in parenthases following each charactarlstic is its mean rating of 
importance to Judgments about quality based on a four-point scale from 
"essential" to "not important*" Under each characteristic are listed the 
indlcaturs that were most highly endorsed by graduate deans as measures 
'j£ that characteristic. The oeans' ratings of these measures are sim-^ 
url^ed on the right side of the table. The first column presents "mean 
adequacy*' ratings based on a four-point scalf from ~*vary good indicator' 
to "rnadaquate indicator*' for all the graduate deans. The second column 
i# the srae s^moary rating computed only from responses by graduate deans 
at hi^^restige universities. The responses of deans from highly 
prestigious universities were tabulated separately in order to test the 
hypothesis that %ome measures of q^llty may be less acceptable in these 
settings than among graduate programs generally. A comparison of the 
mean ratings in the first two columns suggests that deani from prestigious 
inatitutiotis rate the adequacy of most measures very much like ratings 
given by deans in general* If anything, most measures were endorsed 
slightly mora highly by cha deans from prastigious institutions* 

The third and fourth columns on the right side of the tab las repiu- 
sent the percent of graduate deans who Indluated th«£t each measure was 
the one preferred method for measurln the educatloral quality of a given 
program character Is tic. These figures provide a rough index of consensus 
among the deans about the best indicators » though of course the ex ten. 
of thnlr agreement was affected to some extent by tha number of optional 
measures that were presented for each program characr rLstic on the 
original questionnaire* 



"Claffkg M# J. The ^Aasessmant o f Quality In Ph*D . Qgrms ; A 
I Fgelliri^ary Report md Judgments by Graduate Dean ^ « ^i^^. Board Research 
' Report No. 72-7rt. Princeton* N*J* i Graduate l#»cor amlnatlORS 
Boards, 1973 » 
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The last column of figures repres -^s the perceL.: cf all responding 
dewis who said zh^.^ the Information :":piied by the measure was currant ly 
available on their campuses without new data collection, The^e per- 
cantages are probably conservative, ince only Dersans who respc ded 
affirmatively were counted in ths tabulations. However, they tend 
to support the view that a nt oi information about the characteristics 
of progrm participants and program operation should b available frOT 
department or univsrsity records, but that very little irformatlon 
about student, faculty, or alumni actituaes ar.d opinions is currently 
available . 

In sor.e cases, two or more sources of information were possible 

^u^^^^^r, ^- : . * ' , TT-.^-aeur&c rT wp^^ r^ndn^sed bv the deans. 

^Characteristics Tiffed under more than one source ui information have 
heen cross— refer^-'^ ^ed in tbe tables. 

The deans* opinions about Mie assessment of quality In doctoral 
progrMiB guided construci ^on of data collection Instruments it. 
advanced graduate students, faculty members, and recent alunmis as 
-^ell as a protile gl ccpurtincnt acti 'iLi*^- f^nu ^ 4ur--Monnci Lr^ t^ 
department chairmen* 
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Unlvirslty/Dipartittt Raeordi 



U Imimk ttiiolag of faeulty (3 J7) 

Feent with FhiD,'§ oe gqulvalsat de|rge§ 
V of Fh,D, 'i froi "top" pre|faii la neh f laid as 




^ ' '2i ficM^ ryegfch (or oEhir seholirly er efytlvi) aetlvity (3,41) 

ivnigi Buib^ ryyreh proposili fundid in thi last thru ysirs 
pbf fllfieulty, 
» (Alio III Fieulty quiitloMiiEi, ll) 

I 

3i Fieulty (or olhir leholarly or eriitlvs) produetivlty (3,20) 

_ ^Wil|ntid fveragi &per pvbllcatlops (giv^ pro|risslvgly ngri 
i(il|hi to rafiriri jour wis, "liflgli authorihlp of artleliij 
^ 1^ liithorahlp of boiks, Itcj in thg'past thraa ygarg (lub, 
DOi.pitiata or iftistio pirforaaneis er products whin ippfopriiti) 

Aviraii mil journal /.rtldis publishsd In she last thrii 
• yari pir TO faculty [ ' 

. _ 4i GiOttal aeidffllc' ability. of itudents intsjlni the prograa (3,61) 
Ai^i pri enjraduiti iptitudi tilt (GRE^ * 
Avgi uadiriraduate- GPA 

' Avfi 'scon en appropriiti GEE Advinesd fist (or othir leh, tisi) 

■ ' ■ ^ 'k. 

, 5. Ato^ffiti^ IfflpHlldgg, ind/or ikilli of %i^mU it tias of 
^oipletiOQ of dipai (1,61) | 

.'Fircittt of graduatu in thi la§t thrsa yyrs vho obtatoad 
• ■ apkiMBt dirictly filatid to thilr fiild of ipieialiiltion 

Fnent of graduates In thi list 
' prior to tha digrii 

I v*iM III Mui,' quiitioBnaifi, #2) 

ERiCtS; " . . 



Hian 



3.35 3,22 
3,33 3.52 



3,31 3,39 



htiiitr Pircint 

priliErei iViiiibll 

Total fop 50 Total Ts|50 Total' 

im m) m\) mi mi) 



It n 



26 1 



96 

$3 



?4 





1 


• 










74 


61 


21 


3.49 


3.52 


■ 7 


4 


67 


3.42 


3.48 ' 


25 


22 


f 

65 


3,4i 


3,57 


37 


43 


' S4 


3,14 


3.26 . 


9 

i 


9 


,58 


3,21 


3.17 


2§ 


30 




3,2? ■ 


3.26 


21 

1 


' 22 


as 



Hian 



Husufii 

rii 



Pereint 



Titll T§f5^ ^^^^ 



6i hslmisiiii leceipUiliinli d ifiduites (3.38) 

iv|« Ml ^lieatiOBi bjf thsig vho gnduitd In the last 

^ Hsmt of iraduiiis cuf 
miTtfiltiii 

(Aiio 'III Mu. ^uylleaoiire, 13) 

■ r , 

h MviFilly f^iaoiil suppoft for the progfis 
> 8, btmil fiauclii lup^oit fo; the prsirn (3ilS) 



3.30 3*35 
3.12 3JS 



IM 3,00 



§1 lifiiBGUl luppect for ituduta (3.11) ' * 

Av|, dslliri for ill forDi of fteiil aiaiibM»' p« 1^2 
ffliiudttt , . 

iireint @f itudfflti fsciiviiil i!^ fori of fisiBeiil aiiiitiiut 3i2i 3i30 



10, kSn^tim pelldii {Utl 

■ 

himt of plifiid applleiiits vbo m idsittid, ii laporlii 
\il Ihi iMnioB oonittii 

ill IfdviiiiB In thi liiifari of faeulty u^ara (3i42) 

. {^jm Fisttlii quutisnaiirii IS) 
ERIC \" 



3,U 3,10 



23 30 U 
Ik ? 3i 



19 22 



43 
13 



» 22 



1,20 lill M iO* 



74 



OutiUi fisli 11 a psreant sf total prograo budfit 3i42 


3.32 33 3? 


70 


Av|, doUir tocon pir fp fseulty fioi outildi Mureis ovir ^ 


3,43 ' 42 15 


67 


Ebf put ibrii ^uu 






DoUlt umt of fideril reiaateh projicti imU ind contracts ^ 


3,39 12 ? 


79 


ovtt tht ^at tiffti years 







72 
75 



61 



3 



.3 



Hnnl §f lliiii vhd w§li lAo m Ihi Fh,D, 

lilii ^ deetofitii iwfdid to nSi ef 'gfgdyitg fisuitf ud 
10 iiBi of iidUid iiudati 




m M m m .wO ' 



m 


3.12 


41 U 


^' 72 


m 








343 


3.U 


^ 13 a 





33^ 



333 



^ h fiMly fuiiEcb (oi othv WhsliflF or crsatlve) letlviiy (341) 

iviiiii ttiibtf iavitid pria@£itisni\of reinrch rieulli to i 
, ' MM III? (viiltlng lict(ff§e, eo^loquiai «dfkih@ps, 
pfiuli^ iNeiBii, etc) pSF m fai^ylty 

' Itfittt el! ficulti astMy lovolvtd In thg publleitloA of 
. , . ' fiinrGli fiiults (Jsitfaal iditofi dllorial boifd iiaAifii 
. ttfeiifof itibsittid i!tidii, ate,) 
(Alio iH Ihl?,/Depti tmkt It) 



a, 



dUin ilM) 
fisid^ ii|Vei of ap ifflint vith and esmliigDt te thi 



I 



thi,pra|rai 



i -i 



I. 



Fliyily utisfietiOQ with ioflugnci m iipsitant diclsio&i 

Of iipfit (2,?e) 

faeulQNf^P^ed Mtiifactlon with pfograi liadgrihip, 
inEhuaian for £hg pisgran, loyally, InvQlviDiiit, ite, 



4, ^lly d the library (3.60) 
, yipey of rilivint holdingi as judged by im 

k 

i 

5i (]uiUty of laboratory equlparat and laQllltiis (ineluding 
kiUliy for thg eiiativi arti) (3,47) 



j.aS 3,26 

i 



3.3? tM ' 18 22 



3,11 3,1) W 17 



3,32 3,30 42 



39 



39 
39 



3.39 3,09 13 48 



3,48 3,17^ 32 48 



40 



47= 



31 




it ttf^ of enpul^r fieUltiu (1,0?) 

Hlpeif oi esiputii fieiUtiN for niids of thi )t^m 

(Sati^ d ,pf6|rii purpoiig and plisg, y judpd by fieulty 
libiii. 

ifya HI Alti, quistlfiMlrip II) 

8, hs^^ foi Ihi vilfiri of ficuliy m^m (3,42) 

Iiliifi6ti6& iriih frQidn to plan lewsii lad 
sAiuil iMuieh Mitlmt html n sxliTnal iatiffefgaei 



Miki l§r MliSMsci Id nef aod 
- UIn III Doiv,/Dipi/riogrds» fll) 



fieultyi'ii 




f, Frogris ifid^sMp And dieigisn-iikigg (3,3?) ' 

tha pn|7ii 



Quilie; of ludirsldp 



lOi l^lldiiiUps vith eoptg pFograisa (2,94) 

lilillnjhipi and iatiFchan|ei ^th espti proiriig as 
filid ly faculty mhm 



lao if i it 



lyi, #10) 



Spielill QuaUty Htis|i el pro|riiu and faeultiis for e!br 
pio^iy ia thg aaoi field is thle gtudy 



fstii Top 50 
frl7) (8-23) 



3.40' 3,4; 



3,20 1.18 



3.1? 3.32 



m 3.01 



3,39 '3,41 



3.08 3.20 



Heaaufi 



ivailaUi 




25 43 



23 35 



21 30 



3? 43 



U , 21 



.35 



12 



16 
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Htsfi 



diitiimiiiifi 



1. fieidtr &ikm im itydnt dtyiiQpBint aod wilfiEi (3,11)/ 
Avffin ilaiin^ (or fonu i^udoit) ratin| SA a 




fieiil^^naeiiii lor itdmt divilopint a&d/vilfars, accaaii** 
lull;, Mtei ikUli taich^i iffiet^Wiis» §tQ, 
(Alio iM ^yii pitiouiirei ll) 

It Udmt MtiiuAtioB with varioua aapeeN of ^hs dii^sy progran 



3.25 3.( 



(2,89) 



Pif eat of ituirats viw would rieoosend thi progf ii to othira ' , 
vlchjMlir 'loterfits and abUitlie 

i k^t imdiaHatid aitli|actlon i^^h s^kitM iQadisic and 
/ QSD-i£idale lapaots of tk^jfogtai'i 



Oo^uvisi of itudint cartel istiriitB with progru purpoiei 
ai judgid Mf ollid etudinli 



h ' OfQup Wfiii or iiprit (2.S1) 



Studnl iiaai of 



of 



involvaaC is wfEhirhlli aecivltlia, ag fated by studiii 



4« FreviiiGB'fof tk ivaluaiion of atudipt pidgfiaa .(3i41) \| 



1,37 3.13 
7 3,07 

'3,13 3,95 



3.32 

2^/2,83 



^ficwiiile 



0f gfiiyalu (3JI) 



(i^ Ml Mvi/Dipt, 



Iff em to plies griduitii, ai 



(( FfoviiiflB'fdf (hi idviiaat of itudrnti (3.30) 

Qamfef af thi advijjji sjitas^ g§ ratid by mkm 
Mn, quueisauiri, I?) 

h SmdnHMttl^ lalifigEion (3,29) 



utfifaeEloa idth opportusity to wck 
doiii^ }M II Ihie ofli nabir of thi 
(41io 1^ Ai»i .pitiomiifi, IS) 



Si btttfiiUpi 8f fllhif op 
^ ttpriiMU (j;i?) 



lei for itiiivint atudmt 



Mm §^ thi laUnihip or lailgtaQtahlp as fated by 
liydealM ^ hive coiplgted thi Mperi§n§i 
(^N Ni^n, fuutlmiiri, f9) 

h DiifN fiqulfsiati (3,11) 

nixf^ilily of prognB tiqulfninti auffleiatt to Qiit 
\ iadifiduil itudtnt netdi; ai judged by atudents 
\ (Alio III Alum, queitioDsalra, 



Wt liliEleMUpi irttb'cogMti pfe|rMi§ (2,94) 

IlliliofllUps ud IntiEEhaaige vlth cognatg proiriai 
,/ II fitil by liudiBEi 

(AIM III FiEulty pytlssniifip ^10) 

V m 
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idiquiey 
Tetil « Top 50 

im mi) 



m 3.50 



3.22 3.03 



3,11 3.25 



3.06 3.10 



Huaufgi 
FironiC 



3.32 3,36 , 1§ 13 



32 39' 



. 3.3S 3.14 49 ' 52 



40 39 



14 i; 



Utmt 
ivilliUi 




U 



U 



.7 



U 



m 



Chat .irisEic 



L Concgrn for atudint developnient m'h:^ (3 ji; 

> Avirage itudsnt ir fcriner B*:udanE^ ratiLi n a scale inc^SLring 
ficulty concern for gtufi^^t oevalop^ nt m v^lf^qre, fi'-rpq^.l- 
bllity^ advising .HU, teschhg effenivenc"3, etc, 
(Alio lie Student ^estionnairei 11) 

2." AchieveentSj knowlad^^e, and/or skill? f students at tiini of 
coiplition of depsi (3.61) 

Pircint^^f graduates in the k%t three ^'Qars who jbtalned 
ttplopint difictly nhU^ tr their flflii of sp^d.ali^aUon 

Pircint of paduates in the last three years who pubiiahed 
ioiethirig prior to the dearer 
(Aiso sea UnlvJUipt, racordB^ t)j 

> 

Avg, no, publications by tno^! who graauated in th^ hh- 
five year a 

Percant of graduatis cyrrently amployed by doctorace^awarding 
/ universities 

(Also m UniVi/jept. racordSj 16) 

4, Student pirciptions of pru^iM quality, (2*^1) 

Avg, alimni (^5 years) rating of axpiriences in the prograni 

5. Purpsses of the progras (3,42) 

Clirity of prograiD purpoiia and planoj as judpd by recent 
gridiiaEes 

(Also aaa Faculty queationL -raj II) 



ERIC 312 



],V 1.04 I)] 9 

3.28 3,1? 28 30 4? 
3.'7 3,26 21 12 23 

3.30 3,35 23 30 • 11 

3,1? 3,3^ ik q 3a 

m 3,61 68 65, I 

3.29 3.33 11 9 4 

t 

i 



i'^iQint Percent 
prersrred avoilabli 

Total Top 50 Total 



AliMii yuestionnairf 



ftrc III 



ChirictiriiEic _ 

6, PfQvlilon for thi iviluation of student progriss (3 -tl) 

Clirity ef requirenents and standards for progrission fron 
intrincs to candidacy with statid tlnies for rgviiw and 
evaluitian, as jiiGgid by reeent aiimi 
(Aiio lig kdmt questionnairii 14) 

7, ProviiloD for the advlsei: ■'fit of studintN (3,30 

Quality of thi advising systOT, as reated by recent |raduates 
(Alio lee Student quistj 'nnaire, 16) 

8, Student-faculty interictioii (3.29) 

Student-reportid satisfaction with cpportunity to work 
closely with at least one ffliniber of the faculty 
(Also sia Student quistionnairej IT) 

9, IntirMhlps or other cpportunitles tor relfivmit Rtudent 
experigQcei (3,U) 

Vaki of the internship or assiitantsHip if rated by studints 
who have coEpleted the Qxparience 
(Also sag Student questionnairij 18) 

10. Degfii riqulrOTints (3*16) 

FlejElbility of prograi requirciints sufficient to neet 
individual itudent neidSj as Judged by ricnit graduates 
(Also Sii Student qiiistionnairi, 19) 



adequacy 



3.23 3.32 



3,46 3.45 



3,38 3.14 



3,22 3.05 



3.11 3,25 



priferrid ivailabli 



lotal lop 50 Total Top 50 Total 
mi) (H'23) (N-5J) (N-23) (H7) 



25 43 



39 30 



49 



52 



40 39 



14 ' 17 



14 



11 



34i 
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Questionnaires Used In the Study 
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Student Questionnaire A51 
Faculty Questionnaira A59 
Alunini Questionnaire A71 
Depactmental Profilt^ A79 
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GRADUATE SCHOOLS ^ 
IS THE UNITED STATES t 



One pNpnnt Ctfcft: N.W . U hmQion, D C. J(r)i() • Phoni: An.i CoJt J()2: 



J BOYD PAGE 



April, 1975 



Dear Doctoral Student : 



We need your help in a major National research effort to develop better 
ways of describing university programs of doctoral bJudy* Sponsored by The 
Council of Graduate Schools with funds from the National Science Foui dation, 
the project is collecting information from faculty members md doctoral 
students in three departments at ^0 randoriy selected universities across 
the country. Your department is one of those chosen for participation in 
the field study* ^ 

The questionnaire on the following pages wili take about a half-hour of 
your time to complete. Aji enveloDe is provided to insure the confidentiality 
of your responses; your completeu questions ^ it e will be Fj^uer* only by the 
reaearch staff, and summaries of the inforuiation will not lientlfy univer- 
sities by nmme in any repnr*" nf the project, How**var\ yovir department will 
receive a suBsiary of responses made by its doctoral dent i and graduate 
faculty members that can be used for self-study and program improvement. 

The purpose of this project 4S to identify and measure a variety of 
characteristics of doctoral programs that are related to the achievement of 
educational excellence* In addition to the usefulness of such measures for 
program review, they would improve Information available for the guidance of 
prospective graduate students* A detailed description of the project Is 
available from your departme chairman. 

^hink '^cu will ^ir»d the ^ucctisnn^irc intsrsstinQ to snswer and invite 
your coimients on the Itemb as well as on your eKperlences in this program* 

The accuracy of the research findings, and of the sumary report that 
will be returned to your department, 1^ dependent on yuur candid response to 
the following questions* We believe the importance of the study will justify 
the time you give it, Pliaase complete the questionnaire and return it while 
you have it at hand. 

On behalf of The Council of Graduate Schools, I tl mk you for your 
cooperation* 



Sincerelyj 

Boyd Page 
President ' 
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To what extent do you agree with the i illowlng statements about your deparrn^ent 
and its doctoral ptogrms of study? ( ircle one nmnber on. each line. ) 

Agr»e strongly (4) 
^ Agree with reservatlouL (3) • 
ni^afree with reservatiors (2) 1 1 
Disagree strongly (1) | | j 

1 1 1 I 
+ i ^ 

a. The academic demands upon students in this departraent are 

very heavy ^ • * * • ^ - 2 3 4 

* A single "school of thought" or scholarly viewpoint 

dominates this department. 2 3 4 

This dapartmant makes a Gansistent effort to hplp students 

becoise affective teachers * 2 3 4 

d. If 1 had a chance to go to another school without losing 

much in the transfer, I would go.. - 1 2 J 4 

a. This department is providing me with a very good preparation 

for my later rrofesslonal work. * 1 7 3 4 

f. Students in this department have opportunities to influence 

decisions in areas such as admissions policy, curriculum, etc. 12 3 4 

g. It is fairly common for students In this /department to' get the 

feeling that they are being used or exjlofted b:^^ members of 
the faculty p * • 



2 3 4 



h. This graduate program one at tnc best m tne riexd. , , . . . i ^: j ^ 

!• This department has a humane environment characterlzad. by 

mutual respect and concern between students end professors. . . i 2 3 4 

j, Dapar^ental practices create a lot of tension in scudants.- .^1234 

\ 

k, 1 would advise a friend with similar Interasts to come to 

this department, , , , ^ , . * . . 1 2 3 4 

1* There are many opporcunities to take courses in othar 

programs and fields. ....... ^ . * * * ^ - 

skills in this field can ba used to solve social problama 

and Imprpva society - * ' 

n. Students here are highly competent.* 1 2 3 4 

o, I have learned a great deal as a student in this departmant. . * 1 2 3 4 
p. Competition among studants Is encouraged in this department. * 

q. Students here tend to support and help each other meet the 
a^adraic demands of the program. 

r. The department does a good job of piafcing its graduates in 
appropriate positions* . . * * 

s. Many faculty members do not prepare adequately for their 
\^ graduate courses 



12 3 4 



12 3 4 



12 3 4 

12 3 4 

1 2 3 4 

12 3 4 



This department is a stimulating and exciting place to study., * 1 2 3 4 
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How would you rate your department's faculty, facilities, and prograrns? If an 
item listed below doesn't r pi to vour field, or if you feel you cannot respondj 
skip the item, f Circle one number on each line.; 

Excellent (4) 



Fair (2) 

Poor (1) i 



2* Characteristics of the Graduate Faculty 



1 



a* Accessibility to students .... 

b* Helpfulness to students in identifying sources of financial 

. assistance * • 1 

c. Useful criticism of your work . 1 

d. Concern for your professional develupment 1 

e. Guidance in the selecfion of courses • . 1 

f. Willingness to go out of their way to help you If you ask 

f U 4 i L ^ ^ . , , * * * i 

g. Helpfulness in dealing with classwork 1 

h. Knowledge of the field . 1 

1* SchQlarship or research excellence 1 

Exclcemfent for new ideas in the field 1 

k. Overall quality of teaching * 1 

1* Interest in graduate students' welfare , including their 

personal problems . . • . • , , , i * , , 

m, Helpfulnass to students in finding appropriate jobs after 
graduation .............. 

n. Appropriataness of teacihing methods (e.g.* seminars, 

lectures i tutorials, au^io-vlsual aldsj etc*) . * . , . * 



2 

2 
2 
2 



2 
2 
2 
2 
2 



3 
3 
3 
3 



3 
3 
3 
3 
3 



4 
4 
4 
4 

4 
4 
4 
4 
4 
4 



12 3 4 



3 4 



3 4 



3. 



a* Library boiilngs 

b. Laboratory facilities • , . . 

c. CompULur tacliicies . , . 

d. ClaSbiroomH and other inat i uc ^ iunal space 



Excellent (4) 
Good (3) 
Fair (2) 
Ponr (1) 



U 

1 2 

1 2 

1 2 

1 2 



3 
3 
3 
3 



4 
4 
4 

4 
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Good (3) 
Fair (2) 

Poor (1) ; 



I 



Degree Requirements and Curriculum 

a. Clarity of stated rtQuir^iriSuts loi 

b. Relavance of adniisslons requirements to degree requirements . 

c. Agreement between degree requirements and the stated 

objectives of the department 

d. Relevmce of courses in related fields to meet degree 

requirOTents \* * * ^ ^ • * 

e. Appropriateness of language requirements , 

f* Opportunities for independent study ^ 

g. Flexibility of pro^.ram requirements to meet '.individual needs 

Logical st^ucture or sequence of courses . , * * 

^, Fairness with which au^llrvins examinations (or equl^^H] ent ) 

are adminlstfei. ed » . ' 

j. '^"eadth of course and program offerings . . . 

Depth of course and program offerings , * 

1. RelevLice of requirements to anticipated work in the field 



1 

1 

1 
\ 

1 

1 
1 
1 
1 



2 

2 

2 
2 
? 
2 
2 

2 
2 
2 
2 



3 
3 

3 

3 
3 
3 
3 

3 

3 
3 
3 



4 
4 
4 
4 
4 

4 
4 

4' 



Kef 1 el L i t'l is 



Assist ant ships (teaching or research. NOTEi skip these 
i^ems if you have not held an assistantship in this 
department • ) 

a. Number of a^sl^ tantehips available in the department 



Lxcellent (4) 
Good (3) 
Fair (2) 
Poor (1) 



b, Stlprad or payment '^"■r asslstantahlps 

Abstnee of unnecesaary "dirty work" in duties asBlgned 

te assistants • 

Ejctent to which duties contribute to studant academic 

development ■• ■ 

e. Chance for the assistant to act in a profesaional role . . . 

f , Relevance of work to ultimate profussional duties 

g". Degree to which staff members treat assistants as colleagues 

h. Fait and equitable administration of asslstantshlps 

1, Supervision of persons on assistantships 

j. Office space an^d equipment for student assistants 



1 

1 

1 
1 
1 
1 
1 
1 
1 



3 

3. 

3 

3 

3. 

3 

3 



4 
4 



2 3 4 



4 
4 
4 
4 
4 
4 
4 



ASA 



Aboi::. how many of the doctoral students in your ciepart-enr are cnaraci er i ^t^;! 
eac:. of the follow^i^ statements^ Base vour jiirierner .^ on the doctoi-l ^zudt .p 
you have known or observed in the last thre years (or sinue you enterea thiH 
department). :ircle one nuniber on each l^.ie. ) 

*''ore than 7 5 Dercent (4) 
50 to 75 percent (3) 
25 to 50 percent (2) | 
Lctss than 25 percent (1) 



1 i i 



p* They are heavily dependent on direction from the faculty, 
q. They fail to complete major assignments on time*. , * . , 



They work h^rd to meet the intnllsctual demands of the 

department 1 2 3 4 

They demonstrate original ways of handling research or 

scholarly proble-ms, ...1234 



12 3 4 



They share ideas and support one anouher in c-cadem^.: work. ... 1 2 3 4 

They pursue subjects by doing a ^c-d deal o^ unassLgned 
reading p 

e. There is a strong sense of corpetit^'nn among them 12 3 4 

f. They handle assignments with care and respoi -'bility .12 3 4 

g. They are friendly and supportive of une another in 

their personal lives, ......... , , . 1234 



h. They become distracted from their studies by non^ 

academic or non^prof essional interests. , 1 2 3 4 

i. They demonstrate enthusiastic involvement with the field 

during informal discussions with faculty and students,, ,.,1234 

ji They present their Ideas In clashes, seminars j and papers 

in a poorly organized and disjointed fashion. ,1234 

k. They are coOTnlted to the pursuit of scholarship 12 34 

1, Their classroom comments and discussions are interesting 

and thought provoking ^ ' 1 



3 4 



They show imagination and originality in presenting or 

Leauhlug a traditiOAally dull topic. ............. 1 

They continue to work on projects until they are 

successfully completed, despite one or more setbacks 12 3 

They offer weii-founded and consLructlve urlLlulsui^ 

of other students' presentations 



12 3 4 
12 3 4 
12 3 4 



s. They maintain a high level of intellectual honesty 

In their work 1 2 3 4 



i 

s 
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Some programs give primary attenCion lo the pr-paration of schol--^ and 
researchers, others lean more toward the pr^paratton of college teachers, and 
stU^ others give emphasis to preparing practicing professional^ for applied 
and ^-rvice positions (e.g., secondary school teachers , museu- curc^^ors, 
clinical psychologists, industrial chemists). How much importance c you 
think your dapartm^nt now anDigns ' 
(Circle one number on ^ach line. ) 



QS9 t^^^^ different purposes or functions? 
u r TBQ of Import an ce 



a. Preparinf scholArs/reHe^ c-hern 

b. Preparing college Ceachers 

c. Preparing other practitioners 



Extreme 
5 
5 
5 



^^nslderable Some Littla None 

4 3 2 1 

4 3 2 1 

4 3 2 1 



Now- please answer the same qi-Ptinn. in t^rms of your personil goals. How 
much importance do jou assign to these thr^e different purposas or funcrlons? 
(Circle one number on each line.) Degree of Importance 

Kxt reme Conside rable 



a. Preparing; schoLLrs /researchers 

b. Preparing cullugc teacher^ 

c/ Preparing other practitioners 



4 
4 
4 



Some 
i 
3 
3 




he abl^ to analvze our results as mean 
a few things about you. Piease cliecK one 
Che following items. 



ingfully as possible ^ we wouad like to loiow 

legponse liii in the b-ank for each of 



9. In what year did you receive your 
unfjargraduate degree? 19_ 

10. In what year did you first enroll 
in this graduate program? 19 



14, 



13. 



11\ 



12. 



Tfi whrnf ypAr Ho you exnect to 
receive your doctorate? 19^ 



What Is your present degree goal? 

( ) Ph-D* ( ) Ed,D, 

( ) D.A* ( i Other (what? 



13* 



Have you done graduate study at 
some other university? 



( ) Yes* in the same field 
( ) Yes, in a different field 
( ) No 



Are you currently enrolled full 
or part time? 

( ) Full time ( ) Part time 

Approximately what overall grade 
average did you receive in under-- 
graduate aoheol, and what grades 
havtf you received so far In 
graduate study? (Important: If 
your collegp did not use letter 
.uade3™A, B, C. etc. —please mark 
uhe letter grade that is the 
closest equivalent to your grade 
average. Circle one number in 
each column.) 

Graduate grades 
Undergra^H:e grades ^ 

(2*49 or below) 1 1 

2.99) 2 2 

3.24) 3 % 

3,49) ' 4 4 

A to A+ (3.50 - 3.99) 5 5 

All A's (4,00) 6 6 

{lo grades 7 7 



C or lower 
C+ (2.50 - 
B (3^00 - 
B+ (3.25 - 



ASS 



job do you have? you have inore 

than one Job, u^irk Cha one that 
takes .ha most time. 

C ) I am not emp i 

( ) Research asf 4u^ntshiD or 

equlvaleut 
( ) Teaching asslstantsh . ^ or 

e^^ivalent 
( ) O'her university employment 
( ) fisployment outside the unlver-- 

sity in a job relevant to my 

studies 

( J Employmant outside the unlver-- 
sity In a jcb not relevant 
to my studies 

If you are CTployed, about how 
many hours each wee% do you 



( ) 1 am not employed 
( ) 10 or less ( ) 
( ) 11 - 20 



( ) 3l or more 



What kind of position do you h^ya 
to hold on compieL ion u; ^tduurtL*^ 
schaoi? If you are considering 
more than one, mark one first 
preference* ' 

( ) Poatdoctoral fellowship 

( ) Teaching or administration in 

el^entary or secondary 

school 

( ) Teaching In junior college 
( ) Teaching An a 4-year college 



( ) Ui^versity research and teaching 
( ) Coilege or university 

atteilnlstrfttioti 
( } R^iiearch in industry sr wlwn 

nonprofit organization or 
* Institute 

( ) Self-employed professional 



( ) 



f ^ 



Professional practice wifh a 

clinic, hospital, or agency 
i^xecutlve position (adminls^ 
trator, curator, etc) in a 
nonacademic organization 
including government 
( ) Other (Specify: 



23. 



24. 



lat is your a^e^ 



( ) 



/ou a United States citizen? 
[e^ ( ) No 



22. How do you describe yourself? 

( ) American Indian or Native 
rm "lean 

( ) Black, Af ro^American or Negro 
( ) Mexican^taerlcan or Chicano 
( ) Oriental or Asian^American 
( ) Puerto Rican or Spanish-- 
speaking toerican 

( ) Other 



In what university are you 
currently enrolled? 



Indicate (a) your field of 
graduate study wd (b) the one 
sub^speciallEatlon with which 
you identify most clos»ely* 



(a) F 



(b) S^ icic^iizatlon 



( ) c 



HmilSTRY 
( ) Analytical 
( ) Biorhemlcal 

( y HISTORY ' 
( ) Ancient 
( ) Medieval 
) Modern 

( ) PSYCHOLOGY 

( ) Educational 

( ) Measurement 

( ) Personality 

( ) Developmental 

( ) Experimental 

( ) Organizational 



(b) 
I 

( ) Organic 
( ) Physical 
^ > father 



( ) American 
( ) Third-world 
( ) Other 



( ) Clinical 
( ) Social 
( ) Other 



2^. We invite your additional coninarLs about your ^epar^:menL as a place for 
doctoral stud:" Your reacticns to this questionnaire as a means of 
svsteniatl =:al iy ^ollecting rhc^ opininn'^ graduate ?t:udent:s about their 
programs arfc a' ^o welcome, I}iank y.^u :cr juiriplet in^fc the queat ionnaire , 



Plaase seal your completed questionnaire in the accorapanying envelope 5 put your 
name on the outside of the envelope, and return it to your department 01 rice 
(or follow other instructions that may accompany the questionnaire ^hen it is 
distributed). There is no name on the questionnaire; we ask for your nainfj on 
the envalope only that we can tell who has completed the form and who lus not 
The questionnaires will be removed and the envelopes discarded after they have 
been returned to tlie reaearcht»rs , 
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THE COUNCIL OF 
GRADUATE S( HOOLS 



IN THE UNITED STATES 



Dnpont Circle, S.W"., W^ajhiPi^^oN, D L 



* J. BOYD PAGE 
Presi.ient 



April, 1975 

Dear Professor: 

We need your help in a major national research effort to davalop better 
ways of describing university programs of doctoral study. Sponsored by the 
Counell of Graduate Schools with funds from the National Science Foundation, 
che project is coll ci . Information from faculty members*and doctoral 
students in three departm* its at 20 randomly selected universities across 
the country. Your department is one of those chosen for participation in 
the field atudv* 



The questionnaire on the . allowing page^ will take about a half hour of 
your tijie to comple;.e. An envexjpe is provided to insure the confidentiality 
of y^ur responsss ! your =;c^plL2tQ:^ ^u; jticnruiirc be seen only Y,/ the 

research staff, and suOTnaries of the infomation will not identify unlver- 
slties by nme in any report of the project. However i your department will 
receive a sifflmiary of responses made by its doctoral students a^id graduate 
faculty ambers that can be used for self-^study and program improveiaent . 

The purpose of this prtijert 1,^ to identify and meei^ure a v-^^iety of 
charactarlstlqs of doctoral programs that are related to the achleveBient of 
educational excellence* I addition to the usefulness of ^uch measures 
program revlekr, they would improve inf orTtiat ion available for iha guidance of 
prospactl\'e graduate students* A detailed descrlptJ'?n of the project is 

^Vd4,X^U^^ 4 lUUi ^UUL ue^^l LiU^ii U la ^ L Ut^Il ^ 

We think you will find the questionnaire interesting to answer and invite 
your c<:iwmeats on the items as well as on your experiences in ^his program* 

The accuracy of the research findings, and of the siommary report that 
will be returned to your department, is dependent on your candid response to 
the following que^tlnn??. h^li^v^^ th^ in^'TCSncc of tho study will justify 

the time you give it* Please complete the questionnaire and return it while 
you have it at hand. 

^on behalf of the Council of Zrr,^..,ate 3chools, I thank you for yuur 
cooperation. 



S li. "erel y 




CGS FACULTY QUISTIONNAIRE 



To What extent d© y©u agree with the following gtatements about your department 
nd it« doctoral programa of s^udy? (Circle one nimber on each line.) 

m 

Agree strongly (4) 
Agree with reservations (3) 
Disagree with reservations (2) 
Disagree strongly (1)" 



a. The academic demands upon student? in this department are 
very heavy, * 

b Faculty memb^?rs in this department se m to be genuinely 
Interested in the welfare and personal developtnent of 
graduate studer?ts* * , » 

c. Different peisonal It les and scholarly points of view are 

walcome in this department* i 

d. The conflicting demands of my job are a source of considerable 

personal strain^ , , . i * , , , * t # . * • • • * < 

There is wide faculty involvement in important d^artmenta^ 
decisions. * ' ' ' 

f. Some professora in this department unduly exploit their 
students to advance their own researchi 

I hardly ever get the time to give a piece of work the 

attention it deserves* * * ... t ' 

^1. My personal views about graduatie education in my field are 
i ..yiui Mhlc with Lhc Dbjccrivcs and procedures of 
this department. ' * 

1, If T had a reasonable offer^ I would move to another 

u liversity. * . . . . # i . # * • * 

j. Faculty in this department feel very secure in their academic 
freedom to conduct courses and research without undue 
Uepartmental or unlverslLy restrictions.* 

1, t^^^.n*-.^ fir^A ^f-nr^oT^t^^ nf^pf; arrange courses and projects with 
persons in other departments. ^ « 

1. Faculty here often try to apply their knowledge and skills 

in solving social problems and improving society* . » . • - 

.1. Tne department has a humane environment characterized by 

mutual respect and concern between students and fagA^ty* * , 

n. This department is really a coliection of IndlvijMals; very 
little in the way of truly team or joint effort takes 
place in this department, 

o. This department tends, to put more emphasis on training 

technically competent craftsmen than on training idea- 
oriented scholars 

p. In generals I am satl^sfied with my opportunities to influanre 
departmental policies and decisions.. 



1 

1 
1 
1 
1 
1 
1 



if . if if 



2 
2 
2 



2 
2 

2 
2 

2 
2 



3 

3 
3 
3 
3 
3 

3 
3 

3 
3 



4 
4 
4 
4 
4 
4 

4 
4 

4 
4 



• • 9 



1^34 
1 2 3 4 

12 3 4 

12 3 4 
12 3 4 



CopyrlRht- © 1^75 Iducattonfll TestlOB Ssrvlc*. Ail tHhti r#i«rv#d. 



How would you rate each of the following aspects of your department or 
university? (Circle one number on each line.'! ^ 



Ex client (4) 
i;.,od (3) 
Fair (2) 
Poor (i; I 

. . 12:4 



a* Thm Intallsctual envlroni*ent at your university 

b. The Intellectual environment in your department 

c* Personal relatione onong the farultv In your deparLment , . . # 

d, Pscuity/ student relations in yoar department 

a. Clarity of doctoral degree requirfments In yonr department * . 

f* AgreeTient between degree requirements and the stated abject Ives 
of your department * * i * * * . • 

Fairness ^wlth which degree requirements are adulnistered . , . 

ht University library holdings relevant to your field . , t . . * 

1, Laboratory or other equipment needed for ts Ing and 

research in your field . . * - * * • ^ . * * • * ^ • * • * 

Computer facilities to meet the needs in your field 

k. The relationships between your department and cognate 

departments in the uaiversity , , * 

1, Clarity of your department's objectives and plans for the 

next few years ^ , , * • * • 

m* The scholarship and resf-^fch ability of the department 

faculty ^ 

n* Overall quality of teaching at the graduate level in your 

department 

Department procedures for evaluating araduaLe student j^rogresf. 12 3 4 

\p. Department procedures for evaluating student comprehensive 

examinations , ^ . . * * 

q. Department prd^^ures for dissertation supervision and defense. 

r. The scholarship and research ability of students in your 

department , . * , 

s« The "fit'' between youi progrmi goals and standards and the 

needs or interests of current students * - 

t. Coheslveness of the doctoriii curriculian 

Departmental helpfulness to students in finding appropriate 

Jobs after graduatfon • 

The administrative management of your department 1^34 

2 3 )4 



u 



12 3 4 

12 3 4 

12 3 4 

12 3 4 

12 3 4 

12 3 4 

12 3 4 

,1 2 3 < 

12 3 4 

12 3 4 

J* 

12 3 4 

12 3 4 

1 ? 3 4 



12 3 4 

12 3 4 

1 2-3 4 

12 3 4 

1 2 3 4 

12 3 4 



v* 



Departmental effort .toward the career developmet^t o^ ^ 
junior faculty 

Overall adequacy of physical and financial resources, for 

a doctoral progrm in your field , * * . - - . - 4 » • • • * 
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3. About how many of the doetoral students in your department are characterised by 
each af the following st/atements? Base your Judgtnents on the doctoral itudents 
you have known or pbservad in the last threfe years (or since you entered this 
dapartment) . (Circle one number on iach llneO 

More than 75 percent (4) 
50 to 75 percent (3) 
25 to 50 percent (2) 
Less than 25 percent (1) i 

i 1 



a. They work hard to meet the intellectual demands of the 
departnents i^^. ^..,.9. #««•.«. 

t . They demonstrate original ways of handling research or 

scholarly probl^s. , 

c. They share Ideas and support one another In acadeialc work* 

d# They pursue subjects by doing a good deal of unasslgned 
reading # . . 



1 
1 

1 
1 
1 



e. There la a strong sense of competition among them,, * , . * . 

f . They handle assignments with care and responsibility 

The are friendly and suppurtlve of onti another ifi 

their personal lives, • « • 1 

h. They become distracted from their studies by non^ 

academic or non^prof esslonal Interests, , , * * * • 1 

Is They demonstrate enthusiastic involvement with the field 

during infarmal digcug^iuii with faculty and students. . . , • 1 



j. They present their Ideas In classes, seminars, and papers 
in a poorly organized and disjointed fashion, , , « . « 

k*" They are commlted to the pursuit of scholafrshlp, . . * . . 

1, Their classroom con?ments and discussions are Interesting 
and thought provoking,. . . . . , 



m. 



n* 



r i 



They show imagination and originality in presenting or 
teaching ^ traditionally dull topic 

They continue to work on projerts untjl they are 

successfully cumpleted, despLte one or more setbacks. 

They offer well-founded and constructive criticisms 
of oth^r students' prrsentatlons, . , . . , * 

rhey.arp heavily dependent on direction frof^ the faculty 

Taey fail to complete major assignments on time*", * , * 

They maintain a hic,h level of Intellectual honesty 
in their work. 



1 
1 

1 

1 

1 

1 
1 
1 



2 
2 

2 
2 
2 

2 

2 

2 

2 
2 

2 

2 

2 

2 
^2 
2 



1 
3 

3 

3 
3 

3 

3 

3 

3 
3 

3 

3 

3 

3 
3 
3 



4 

4 
4 
4 
4 

H 

4 

4 

4 

4 

4 
4 
4 



3 4 
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4* Three gra^ral argas of emphmsla descrlba most doctoral-level grmduate 
prograns* Soma programs give primary atterstlon to the preparation of 
scholars rad rpsearcheray others lean more toward the preparation of 
collegs csaehsra, and still othars give emphasis to preparing practlGlng 
'profMSloaals for appllad and sarvlea poslclons (e,g#, aacondary sohool 
tMcl^rai mammm euratoss, cllnlaal psychologists » Industrial ehtaists). 
How Tweh li^ortanee do you think your departaent now assigns these three 
dlffaraat purpoaea or functtsne? (Cltrle one nuTsber on each line.) 

Degr aa of Importance 

Extreme Conildarable Soma Llttla Hona 

a# Praparlng scholars/researchers 5*4 3 2 1 ' 

b. Preparing college teachers '5 4 3 2 1 

c. Preparing other practitioners 5 4 3 2 1 



5* How much Importance do you think the department ought to assign to each 
of these purposea? (Circle one number on each Una*) 

Depree of Importance 

Extreipe Conilflerabla ' Some Llttla Nona 

^« Preparing seholars/reiearchers 5 . 4 3 2 1. 

b. Preparing collie teachers 5 ^4 ' 3 2 1 

Ci Preparing other practitioners 5 4 3 2 1 

6. In /a typical week during the academic yearp about how many hours du you 
spend In each of the following activities'^ (Please account for all of 
your professional tiroes) ' 

Nmber ^ 
of houta 

a. Teaching, preparing for classes,' evaluatlfig student 

tests or papar^, 

b. Advising students and directing students' research . 



c. Research and gcholarly writing 



Prpgram or university administration, (including cOBplttees 



li^ees) 

a* Private practice V 
f. Other professional activities Including consulting 

Total I 



How much of your time in (a) and (b) above (teachings advising , supervising 
research) Is spent with or for doctoral fitudents? ' 

( ) more than 75% ( ) 25 - 49% 

( ) 50 - 75% ( ) less than 25% 
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8, Please clrela the approprlat* response to indicate your experience in 
Che following areas; , - 

• ' ' No (2) ' 

. , . . Yes (1) I 

a* Have ym umrvmi on govarment or foundatioQ ravfew eonslttaaa, 
alca Vlllt tasns or tlatlonal ad's^sory aounells In the last 
I thifaa yaara? 1 2 
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Hava you hald a postdoctoral fallowship? * * 12 

a, FriVa you rs^aivad a sarlous Job inquiry from anothar 

«ployM^ In tha last two yaars? ^ 12 

d* Have yau raealvad an award for ouiistandlng teaehlng? j 1 2 

mm Have you racaivad an award for outstanding resaareh or 

scholarly writing? 1 2 

f . Have you racaivad an award for outstanding prof asslonal 
practica? « * ^ 
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1 2 



g. Hava you held office or served on boards of a national 

professional association or organisation? ..*»# 12 

h. Have you held office or served on comnlttees of state or 

regional professional organlzt^tions? » 1 2 

i« Rave you bean the ^Itor of a Journal In your field or served 

as a mraber of a professional Journal editorial board? . « . # ^ ^2 

Ji Have you refereed oontributed articles for a professicnal 

Jou^ lal in your field In the last two years? 12 

k. Do you have an institutional or department grant to support 

your research thi s year? * ..•# 1? 

1* Do you have a grant or contract with an agency outilds the 

university to support your research this year?* • * 12 



.9. 
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Tlaaae list the BwAez' of your publications In each of the following - 
areas for: (a) your entire professional career and (b) the last three 
years'. Include Itras written alone or In collaboration with others. 



ProfessloMi srcleles and aingle chapters In books 

Sek3lmrly book reviews ' 

Authored books i 



Edited books or Anthologies 
Monogrsphs and manuals 



. <i>) 

Last three 
years 



10, 



11, 



12, 



About how many flays In the past 
12 months were^ you away from 
OMiipus for professional actlvl^ 
ties (esg. prof essional meetlngSi 
speeches i consulting)? 



How amy times in the past 
12 nonths have you presented 
some of your research results 
or other scholarly material as 
a colloquiim speakerp visiting 
lecturer on anuthar campus , 
speaker at a professional 
meetings etc*? 



What is your preeent academic 
rank? 

( ) Do not bold rank designation 

( ) Profefspr 

( ) Associate professor 

C ) Asslstmt professor 

( ) Lecturer or Instructor 

( ) Othe r 



13 « What is your highest earned 
degree? 

( ) Ph.D, , 

C ) Other doctorate (ld*D. ^ 
D.At , H.Dt , etc.) 

( ) Masters or equivalent^ 

( ) Other 



14. In what year did you receive 
your highest earned degree? 



15. 



Prom what university did you 
receive your highest earned 
degree? 

N^ei' 



State; 



EKLC 
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16. Wrtiac li your present university 

( ) Full tlBtt 

( ) Fart tiM 
X ) Adjwiet 



17. you hatve tmure at thl! 
uDlvstsity? 

t ) No 



18* How m^y years have you been 

assnelatad %rtt\* this depattment? 



21* 



19. D© you hold a Joint appointment 
with anothar depattrQent? 

( ) Yas * 

' ) No 



20* Do you hold a joint appointment 
. with a cafiter or institute? 



( ) Yas 
( ) No 



Bnw Aany years of university 
teaching wperlence have you 



22* t'That Is your sex^ 
( ) Mile 
( ) Femalj^- 



23* In what university are you 
currently teaching? 



24, 



Indieate (a) your discipline 
ond (b) the one sub-specialisa- 
tion with whieh you identify 

most fslosely* 

(a) Qisclpline 
I 

(b) Speciiilization (b) 
♦ i I . 

( ) CmflSTRY 1 



(*) Analytical 
( ) Bioehemieal 
( ) Inorganic 



( ) nil 



i ) Aneient 
( ) Medieval 
( ) Modern 



1 



( ) PSYCHOLOGY 

( ) Bducatlmal 

( ) MttasuretQCTt 

( ) Fersonaltiy 

( ) De^^elopmantal 

( ; Ejcperimantal 

( ) OrganiEatloT.il 



{ ) Organic 
( ) Physical 
( ) Other 



( ) ^erican 
( ) Third-world 
< ) Ott er 



( ) Cliniciil 
( ) Seeial 
( ) Other ^ 
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2? For each unlvtfsifey llittd below, rate the docteral programs in your field (dimclpllne) 
ofi th« following qutPtlons* Your ratings will be used only in research associated with 
this projecti siraafy ratings will n9t be made public, 

QuMt lon Ai Quality of Graduate Faculty . Circle the nianber below the term that cor- 
respondii most closely to your jud^cnt of the quality of the graduate faculty your 
^ field at 'each InstiEution listed. Considar only the scholarly eompatence and achieve- 
Ments of thg prerant faculty « 

Quegtlofi Bs Attractiveness of the Ddctoral ProgrMi . Circle tha number below the term 
that correaponds most closely to the way you would rate the Institutions Ustad if you 
wre seleetlfig a graduate school at which to work for a dQCtorate today, 
J^^ccount the accessibility of the faculty and Its scholarly cempetance, the curric 
^ tha instructional ^nd research facilities, the quality of graduate studantSp and 
faators that cbntrlbuta to the effectiveness of the docto^l program. 



take into 
the curricula, 
other 



QUESTION A 



QUALITY OF 
G^UATE FACULTY 



QUEStlOH B 



ATTRWTIVENESS OF 
DOCtOML FBOGBAH 



INSTITUTIONS 

(A sample of universities 
awarding doctorates in 
your field, arranged 
alphabatlcally) 




Boston College 

tol* at Berkeley, U* of 


1 2 3 4 5 6 7 
1 2 3 4 5 6 7 


Csl. at Davis, U* of 

Cal. at Los togeles, Ur of 


1 2 3 4 5 6 7 
1 2 3 4 5 6 7 


Colorado, U« of 
bory U* 


1 2 3 4 5 6 7 
1 2 3 4 5 6 7 


Florida State U* (Tallahassee) 
Indiana U, (Bloomington) 


1 2 3 4 5 6 7 
1 2 3 4 5 6 7 


Iowa, U, of (Iowa City) 
Kansas, ti* of 


113 4^67 
1 2 3 4 S 6 7 


Louisiana State U, (Baton Rouge) 
Karyland, Ui of 


1 2 3 4 5 6, 7 
1 2 3 4 5 6 7 


Hlchlgan, of 
Himiesota, of 


1 2 3 4 5 6 7 
1 2 3 4 5 6 7 


Hiasourl, U« of (Colimbia^ 
Hew York U« 


1 2 3 4 5 6 7 
1 2 3 4 5 6 7 


Northj^terii U. 

Oklahoma Stat^e U. (Stillwater) 


1 2 3 4 5 6 7 

1 2 3 4 5 6 7 


Faimsylvanla, ^« of 
Princeton U* 


12 3 4 5 6 7 
1 2 3 4 5 6 7 


Stanford U, 
Toledo , U of 


.1 2 3 4 5 6 7 
1 2 3 4 5 6 7 


Utah, U. of 
West Virginia U, 


1* 2 3 4 5 6 7 
1 2 3 4 5 6 7 


Wisconsin, U. of (Madison) 


1 2 3 4 5 6 7 





4J . 
^ / 


r O i 
^ 1 






1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


a 


3 


4 


S 


1 


2 


3 


4 


5 


1 


4 . 


3 


4 


5 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


2 


3 


4 




1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


r 


i 


k 


i 


1 


2 


3 


4 


5 


1 


2 


3 


4 


,5 


1 


2 


3 


4 


5 


1 


2 


- - 1 
3 


4 


5 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 
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26. On this pagti tmtm each univargicy^s doccorai progrm in the axea _Q^t__gjji_CAa chat 
y«u Ittdieated in ttm 24 (b)« ~ , 

Qyartiati Ai Quality of Graduate FaGyXty * Circle the number below the term that cor- 
M^p^adi Boat clesaly to your judgment of the quality of the graduate faculty In your 
ap#eiAlisatioQ at aaeh institution listed. Consider only the scholarly competence and 
aehla^Mcnta 



Qugation Bi 
that eornsponi 
vers •aiccting 
iaceotmit ttia acc^ 
thm Inatruetiona 
factors that cont 
tian. 



INSTITUTIONS 

(A sanple of universltie 
awarding doctorates in 
your field j arranged 
alphaba t ieally ) 




^toaton Collage 
Gal. at Barkaley, U« of 


1 2 3 4 3 6 7 
1 2 3 4 5 6 7 




1 2 3 4 5 
1 2 3 4 5 


Cal. at Davis, U. of 

€al* at Loa Angeles, U, of 


^12 3 4 5 6 7 
1 2 3 4 5 6 7 




1 2 3 4 5 
1 2 3 4 5 


OMorado, U« of 
Moty U. 


1 2 3 4 5 6 7 
1 2 3 4 5 6 7 




1 2 3 4 5 
1 2 3 4 5 


Florida State U. (Tallahassee) 
Indiana (Bloomington) 


1 2 3 4 5 6 7 
1 2 3 4 5 6 7 




12 3 4 5 
1 2 3 4 5 


Imra» V* of (Iowa Cit/) 
Xanaaa. U» of 


12 3 4 5 6 ? 
1 2 3 4 5 6 7 


1 2 3 4 3 
1 2 3 4 5 


liMlalaaa itata (Batou Rot^fe^ 
Mat^laad. U. of ^ 


1 2 3 4 5 6 7 
1 2 3 4 5 6 7 


1 2 3 4 5 
1 2 3 4 5 


' MlehltMi. U« of 
liteaiMdtat U# of 


1 2 3 ^4 5 6 7 
1 2 3 4 5 6 7 




1 2 3 4 5 
1 2 3 4 5 


tttSMiirip U» of (OoluBbda) 
Rm York 


1 2 3 4 5 6 7 
14 3 4 5 6 7 


1 2 3 4 5 
1 2 3 4 5 


Rorthifaatarn 

OklalioM Stata U. (Stiilwater) 


1 2 3 4 5 6 7 
1 2 3 4 5 6 7 


1 2 3 4 5 
1 2 3 4 5 


V«gnaylyaala, U« of 
Frlneatm U. 


1 2 3 4 1 6 ? 

1 2 3 4 5 6 7 




1 2 3 4 5 
1 2 3 4 5 


Stanford U. 
folado. U. of ^ 


1 2 3 4 5 6 7 
1 2 3 4 5 6 7 




1 2 3 4 5 
1 2 3 4 5 


Utah^^ of 
Vaat Virainia u/ 


1 2 3 4 5 6 7 
. 1 2 3 4 5 6 7 




1 2 3 4 5 
1 2 3 4 5 


Viaeonain, U. of OUdiaon) 


1 2 3 4 5 6 7 




1 2 3 4 5 
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27.' Pleas* use this space for any conments you vrould like to make about the 
•ducatlOTsl. quality of doctoral programs In your department or Issues 
eoneened with the assessiiient of quality in doctoral education. Your 
rMcUdns to this questionnaire as a means of syst^tlcally collecting 
the oplnloBS of faculty mmbers aUbut their programs are also welcome. 
Thank you for co^letlnR the queitlonnalre. 



Please seal your completed questionnaire in the accompanying envelope, put your 
name on the outside of the envelope, and return it to your department office 
(or follow other instructionB that may accompany the questionnaire when It Is 
distributed). There is no name on questionnaire | we ask for your name on 
the envelope only so that we can tell who has completed the form and who. has not. 
The questionnaires will be removed and the envelopes discarded after they have 
been returned to the researchers. J 
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THE COUNCIL OF 

GRADUATE SCHOOLS 
IN THE UNITED STATES 



feU, N.W„ Washington, D. C 20036 • Phom.- Area Code 202^ 223^79i' 




T. BOYO'PAGB 
Prtiidmtt 



< June, 1975 

We ae^ your hslp/lti a najor matlanal research effort to develop betcer 
vaya y? deeerlblag UBtversity progrms of doctoral stydy. Sponsored by the 
Cotmell of Gfaduate Schools with funds from the National Soienoe Fomdatlon» 
the projeet la eoll^ctlng Inforaatlon from faeulty m^berspi doctoral atudentSp 
^d departmst alu^ii 1p three departments at 20 randomly gelected univeraltles 
icroas the country* Your former department Is one of thos# chosen for partlcl- 
atton in the t%klA study « 

/■ ' ' ~ 

The questionnaire on the following pages will take a^out a half-hour of 
your tlne-€b complete. An envelope is provided to insure the confidentiality 
^F j^w your cos^leted questionnaire will be seen only by the 

-^search staff » and stmaries of the information will not, identify unlver^ 

ties by n^e in any report of the project. However » your former department 
fill receive a simary of responses made by Its doctoral istudentSi graduate 
^acuity mnbersi and alumni that can be used for self^sti^y and progm 
yi^rovnent. I 
i . ^ , ■ ^ 

The purpose of this project is to identify rad meaifure a variety of 
characteriatlcs of doctural progr^ifl that are related t^ the achievement of 
edueatlonal ^cellence. It addition to the usefulness of such measures for 
program revlewp they would improve information availabli for the guiduce of 
prospective graduate students # / 

think you will find the questionnaire interesting to aoswer rad Invite 
your consents on the Items, as well as your experiances in this progrm^ 

The aceuraey of the researeh findings, and of the su^Mry report that 
1^11 be returned to your department, Is dependent on ^our cmdld response to 
the folloiflsg questions. We believe the importance of the study will Justify 
the time you give It, Please complete the questlotmalre rad return It while 
y^u have it at hmd. 

On behalf of the Council of Qraduate Schools « I thuk you for your 
cooperatlon« 



Sincerely, 



Boyd Pa: 
President 



CGS GRADUATE A]^MNI qUESTlONNAIRt; 

1 To what «t#nt do you agree with th« following at.t^enta about the department 
to whlfen you did your doctoMl study and Its doctoral ^grans of study? 
nJ.! give your opinion bmsed on your beat recollectlM of your graduate 
•chool «perl«ncei. . (Circle om number on each line.) 

i^ree ■trdngly (i) 
Agree with reBervations (3) 
Disagree with reBervations (2) 
Disagree strongly (1) I 



a. A single "school of fhought" or scnolarly viewpoint 

dominated the department i 

b. It was fairly eowwn fo. studenM In the department to get 
' . the fftellng that they were being used or Msplolted by 

members of the faculty 

c. I consider the graduate program one of the besf in the field.. 

d. The departmeht had a humane environment characteriiAd by 

mutual respeSt and concern between students and professors. . 

e. Departajental practices created a Idt of tension in students. . 

£. I would advise a friend with similar interests to 

/ study in this department. • • f ' 

g. There were many opportunities to take courses, In other 

programs and fleldB, • * * ' 

' h. I learned a great deal as a student in this department.. . . ■ 
1. Faculty members emphasized ways in which knowledge »nd skills 

in the field could be used^to solve social problems and 

improve society • 

j. Cooqietition Mnong students was encouraged in the department, 
k. Students supported and helped each other meet the academic 

Hamanriw of the program • 

1. The department was a Btlmulating and exciting place to study. 



•I 



I 
1 

l' 
1 



I 
I 



2 
2 

2 

2 



3 
3 

3 
3 



2 
2 

2 
2 

» 

2 
2 



12 3 4 



4 
4 

4 
4 



2 3 4 



3 
3 

3 

3 

3 

3 



4 
4 

4 
4 

4 
4 



How would you rate your formar department's faculty and programs?. If ".i'?^ 
b^o^ doesn't apply to your f laid, or if you feel you cannot respond, skip the it«. 
(Circle one ntnober on eac' . line ) 

focellent (4) 

Good (3> 
Fair m 
Poor (1) 



2. Characte? .istics of the Graduate Faculty "^-^ 

a. Accessibility to students 

b. Useful criticism of your work 

c. Concern for your prolessional development 

d. Guidance in the selection of courses . i 

^prti^t e *y Wueatleoal f«ae4a| Sawle*. Ml tlihei wwer^wl. 
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Excell^ar (4) 

Fair (2) 
Poor (1) 



Chagmctegisties of the Graduate Faeulty (cohtiriued) 

e» BplpfulaasB lii daaltng ^^th clssswdrk « , * • • 
f. Ibo^l^ga of the f ield 
g* Seholarshlp ot f esea^^h ^cellence 
h, BAthuslam for the field 
1# Overall quglltf of teaching 



1 1 



• • • 



■ > • 
• • t 



• i • 



c ■ ■ : • 



s ■ t t ■ • c 



1 
1 
1 
1 



3 

3 

3 
3 



# 

4 
4 
4 



ji I^tereiit In graduata itudenta- welfare, Inaludlng their 
parsf Jtl probltos 



1 2 



Excellent (4) 
Good (3) 
Fair (2) 
Poor (1) 



Pagree Requlr^iente and Jurrlculim 
a* Clarity of stated rf^.qulreinenti for the degree . 
b. Agra^iant batvean dagrea raqulr^enta and the stated 



1 1 



• • • 



12 3 4 





objectives of the, departDent • 


. . 1 


2 


3 


4 


c. 


Relevance of courses In related fields to meet degree 
re^iilr eifieiit 8 


. . 1 


2 


3 


4 


A. 


Appropriateness of language requirements ■ • • 


. . 1 


2 


3 


4 


m. 


Opportunities for Independent study .... 


. . 1 


2 


3 


4 


f. 


FleaElbllity of progrm requirements to meet Individual needs 


. . 1 


2, 


3 


4 


&• 


Lcgical structure or sequence of courses 


. . ,1 


2 


3 


4 


h. 


Falraeas with which qualifying exmlnatlons (or equivalent) 
wero conducted, and evaluated 


. . 1 


2 


> 


4 


1. 


Braadth of course and program offerings 


. . 1 


2 


^3 


4, 


h 


Depth of course uid program offerings 


. . 1 


2. 


3 


4 


k. 


Relations with related departments in the imlverslty .... 


. . 1 


2 


3 


4 
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Overall, how well do you thlnK your dyartment prepared you for work In each of 

the following areas? (Circle one niMber on eaeh line.) 

( " ■ 

^tr^sly 



I 

a# Scholarly research 

b» Collage teaching 

c. Other professional practice (a.g.» 
cllnlclm« cur^nor, product 
i^nager, etc«) 
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Not very well Fairly wall wall 

1 2 3 

1 2 3 

.1 2 3 

3iia " 



1 
1 
1 
1 
1 

1 
1 

1 

1 
1 
1 



2 
2 
2 
2 
2 

2 
2 
2 

2 
2 
2 



3 
3 
3 

3^ 
3 

3 
3 
3 

3 
3 

3 



fle&ie €vsiumCe the following components of tba dissertation phase of {he 
doctoral prograai In your dfepartmenc according to the icalt at the right* , 
(Circle number on eacn line.) - Satiifaetary as is (3> 

Could be taprovad (2) 
Neads substantial revision (1) | 

a» Incagratlon of dissertation research and course work . 

b. Procaduras for selecting the chairman and comittee members • » 
e. Fraadoia of student to select own topic 

d. Canarally axpacted scope of research pvobln . • . • 

a* Formal sad inforaal Errangements for studei\t/faculty Ihteractlon 

£* The nature of the supervisory relationship between chalrBanp 
conlttee and candidate « « • 

g. The nature and timlAg of ttie final oral e^caninatiom * . . . t 

h. Quality of writing expected in thelfinal document 

1* Opportunity for the. student 's creative thinking and individual 
expression «• 

J. Relevancdlof dissertation research ttf other profsssibnal skills 

k* Relevanc^t^f dissertation research to student's career plmis , , 

As you look back t^ your graduat^^ school years * and considering your prasaiit 
position, how valuable or useful Vere the following aspects df your graduate 
Experience as p jparatioh tot your present work? If a particular eKparlwea 
is not applicable to youj circle t|he number in calu^ Ccircla ohs numbag 
on each line*) * 

Hot applleabls (5) 
' ^ Very great valua (4) 

^onsidarabla valua (3) 
Sopa value (2) 
Very little value (\) \ 

a^ Required courses in department 

b. Elactiva courses in dt^partraent 

c^ Course work In other depart^nents * , , . r 

d. Association with your maj^or professor * 

e. Association with other professors * . , . v 

f. Associatinn with fel - graduate students 

g. Experience of working pn your dissertation 

h. The dapartment's standards of ^cellence for work in the field 1 
1. The cultural md social life of the university * . * 
J * Technical skills learned .in course or research work 
k. Rnovladge gained in course or^ research work «... 
1. EHP^rlence as a research assistant • , . , , . ^ . 
n« Experience as a teaching assir4.ant , * * * 
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■ < . 1 


2 


3 


4 


5 


. . . 1 


2 


3 


4 


5 


.• . . 1 


2 


«^ 


4 


5 


. . . 1 


2 


3 


4 


5 


. . , 1 


2 


3 


4 


5 


. . . I 


2 


3 


4 


5 


. . . 1 


2 


3 


4 


5 


field 1 


2 


3 


4 


5 


. . . 1 


2 


3 


4 


5 


. . . 1 


2 


3 


4 


5 


. . . 1 


2 


3 


4 


5 


. . . 1 


*2 


3 


4 


5 


. . . 1 


2 


3 


4 


5 



o oe BDLB CO analyze our 
a few things about you. Please -her;- 
tha following items 



■U:u:fuJ 1_ -iH 



'■'•i"r; for e-. ul 



1 . What is your s 

( ) Mal^ ( ) : rHn;ale 

8, Wliat is your age? 



9. Are you a United StH citizen; 
( ) Yes ( ) N ; 



10^ How do yo^.i detjCtLb^ 



( ) American Indian c: N.^t:iv- 
AmericEn 

( ) Blackj At L ^^^/^erlcai or Ne^ro 

( ) Oriantal i .^siai: Amer ican 
( ) Puerto Kican or Spr^ ish- 

speaking American 
( ) White or Caucasian 
( ) Other 

11. Approximately what overall grade 
average did you receive in mder- 
graduate school ^ and % .it gra^-Ias 
did you receive in g aduate study? 
(Important^ If your collefc^ did 
not use letter prades~A, B, C, 
etc,=-please mark the letter gfade 
that Is the closest equivalent to 
your grade average. Circle one 
number in each lolumnO 

Graduate grades 
Undergraduate grf.des 

C or low*-r (U^H or h^l™;^ 1 

C+ (2,50 - 2,93) 2 

£ (3.00 - 3,24) 3 

(3.25 ^ 3,4y) ,^ 4 

A to A+ (3,50 - 3.9Q) 5 

Ml A's (4,00) 6 

No grades 7 

L2, In what yirar did you re elve 
your doctorate? 



1 

2 
3 
4 
5 
6 
7 



14. 



'rHicace (a) your field graduate 
cr^^r^> ^tia (bj Lhe one sub- pecialiZ'U= 
t ion w^t;h .■''ich y-u idanLify Taost: 

^'a) Fi^Ld 
i 

i (b) specialization (b) 



( ) Crganic 

( ) PHysic^l 
( Other 



i 

( ) CHmiSTRY 

' ) Ajialyti^ al 
C ) Biochemical 
( ) Inorganic 

HISinRY 

.. I . . . 

( ) Medieval 

{ ) I YCHuLQGY 

( ) Eduratin al 
( ) Measuremer.: 
( ) Person^^lity 
( ) Developmental 
( ) Experimental 
( ) OrganiEational 



How helpful were aacn of the follow^ 
ing in finding a job for you when 
you completed doctoral study; 
(circle one number 

on each lln^,"! Vpry b^lpfnl (1) 

Of soma halp (2) 
Not at all helpful (1, 



( ) Third- ^wor Id 
( ) Other 



( ) Clinical 
( ) Soc^ 1 
( ) Otht 



My departrnent ' s formal or 
informal efforts ^ , , 



The assistance of 

individual piofessorG 

The university piacemeiit 
office . . . . , . 



12 3 



12 3 



1 2 J 



Openings listed with 

professional asHociation 

Letters sent directly to 
profipectiv! empi.jyers . , 



12 3 
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How wou= 1 you aescrxDy ixis^ 
position you held after completing 
all requir Tients for \ our docLorat-= , 
and how would you describe your cur- 
rent position? If more than one stat^-- 
mant descr--btS the posir.i mark the 
ne that was most time consuring. 

First pcjitiun 

Current position 

( ) (^) Postdoctoral fellowship 

( ) ( ) Regulai academic appointment 
at a PhD^grantin^ imiversity 

( ) ( ) Regular academic appointment 

at a four^year rnll^^Re or non^ 
PM-g^antin<^ uuiverF 

( ) : ) Lecturer or adjunc; ^a.-ulty 
apnointmen' at a fovir-ve^Rr 
college or university 

( ) ( ) Adminisuratl^ J r-siticn at 

-four-year '.DlLegB ^ r uiTiversity 

) K ) Jommunity co ^ L.age position 

( ) ( ) Prln 4ry or secondary school 
position 

( ) t ) Reaaarch position with academic 
or other nonprofit agericy 

( ) ( ' Research position in business, 
j^adustryj or government 

( ) ( ) Administrative position in 

business, Indus tryV' o^ gjvern-^ 
ment 

C ) ( ) ?rnfp^^innal practlcp in a 
clinic,, agency or hospi il 

( ) ( ) Self-employed or private 



( ) ( ) Continuing graduate or 
professional education 

( ) ( ) Not employe'd fur pa- 

( ) ( ) Ot^r positiun (specify) 



il 

Which statement best describes how vou 
regarded your first jot after leaving 
gr^^duate school, at the time you 
accapted it? ^(Mark one,) 

( ) Job to earn money while I looked 
for something else 

( ) Job with p^sible carler potential 



stater^unts for the accuracy with 
which they describe your current 
rof^itlon, (Circle one number on 

^^^^^ = vary descriptive (3) 

noRcripiive, with reservations (2) 
Not descriptive (1) 



It provides a very com-- 
fortable salary , . , . 

It provides many oppor- 
tunities for research 
and creftive \ 5rk . . < 



1 2 3 



Q . 
P . 



It does not use all my 
education and kills . , ♦ . 1 

It is a ver/ seCur^ job . , 1 

It provides good oppor^ 
tunic ies ^or advancemei;^ 

It allows me to increase 
my visibility within the 
profession 

It innerferes wltlj ..the 
preparation of articles 
qnd/or bonk^ • . 

My colleagues are first-^rate 
scholars or .esearchers * . 

It lef^es me relative. 



3 
3 



12 3 



12 3 



12 3 



free 

> 



k. 



12 3 



12 3 



1 
1 



2 
2 



3 
3 



IB, 



of luperviFlon by otht^s 

It givec me a chance to 
exercise leadership , . . ■ 

It proviites maas opportuni- 
ties to be'Helpf^ to 

Host days I enjoy It . * . ^ 

T gives me a feeling of 
wurthwhiie accomplialimer . 



\ 

What 1^ your prima j-y acti^^lty in you" 
present job? (Cuagk one,) 

( ; Research 

( ) Research and teaching 

( ) Teaching 

( ) Administration 

( ) Otner professional practice in 
the field 



( ) Job ♦with definite career potential 
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( ) Other 

371 



( ^art time 

If you hold an academy ^ position, 
what 1^ your present 'academic rank. 

( ) Do not hold rank designat ion 

( ) Professor 

( ) Associate professor 

{ ) Assistant professor 

( ) Lecturer or Instructor 

( ) Othe r 



prnf^ssional activities (e.g,* 
honorarit. : royalrJ i) . Incl.uie 
Balary befcre deduction's but witho-it 
fringt: tninfc ' ts paid '^.y che err: ploy t, 
(Check one . ' 



( ) Less Lh^in S1U,0'J0 

( ) $.j,000 - 314,999 

( ^ $15,000 - S19,999 

( ^ S20,000 - 024,999 

( ) $25,000 - $29,999 

( ) $30,000 - or abovp 



If you hi Id an academic positiun, 
how would you describe your 
prospects for tenure? 

( ) Good 

( ) Uneure 

( ) Poor 

( ) Very poor 

Since completing your dissertacion, 
have you done subbequenL research 
In the BBme area as that of your 
dissertation? ( ) Yes' ( ) No 



Have you published any part of your 
dissertation? ( ) Yes ( ) No 



To what ext mt is your job related 
to your graduate field of study? 

( ) Not at all related 

( ) Somewhat related 

( ) Highly related 



Do you cotisider yourself under^ 
employed In your present position 
beeause It is not in 'your fifeld 
or not consis ent with your le\ il 
of training and experience? 

( ) Yes \ 

( ) No 



27. 



28. 



29 



30. 



Please list the number of your publi^ 
cations ci presentations in each of 
the f i . lowing areas* Include items 
.'Ver Hrp i alone or in r ol labor ^ ■'^n 
with orh^ars* 

P u bl lCj . t ior I. Number 

Professional articles and 
single chapters in bc3};s 

Authored books 
Edited books or anthciogies 
Monographs and manuals 

P r a s en tat ions 

At regional national 
prafesalonal meetings 

At scholarly Golloquia at 
own or other institution 



DM vQu ever takfi the Graduate 
A*curd ExMlnations? 



( ) Yes 



( ) No 



If yes 5 would you he willing to 
release you% scores for research 
purposes? 

( ) Yes ( ) No 



If y^^t -t^hat is your best racol- 
L "tion of the month and year when 
yoM last took the test? 

Month Year 



f2 



doc > oral study* "-ur r<^iLt;on^ 



col let ing 



ne 5f 



- prog: also 



Plear.-J seal your completed q ies l io mair in the accomp. nvtng envelope and drop 
it in the mail* Thank you far com letxng the questionnaire* 



EKLC 
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3 7.8 



Note : Clrc' d items we:e evaiuatet^ 
program indicators for use in this rf art. 



Univ, :^ 

'^nrHnn completing this ropc^rt i 



The follcving questlom are deal, aed co provide an yvervit^ uf yuur ^uctoral progran^.s. 
Unlass speelfled othervise, the questions refer only to doctoral-level training and 
research, not undergraduate or master's decree programs. Please v^ite in the appropriata 
ni^abers for each question in the spaces provi ed, and add coOTients at the end of the fom 
to explain unique or utiusAl aspects of your prograffl that snould be taken into considera- 
tion when InLerpi^cing rte data, 

1. FACULTY STAItJNC 

/ \ 

^\What the total number of faculty meiLbers instructor through professor) in 

yoiir department In Fall 1974, to teach graduate and undergraduate students? 



Full time 



Part time 



tTE* 



bT^What is the 1974-75 average full-time salary fur each faculty rank, not including 
fringe benefitSi for a 9- ir lO^month appointment? 

^ Professor Instructor J 

All faculty 



Associate professor 
Assistant proressor 



c. How many full-time faculty have left yc jr department since July, 1973?^ 



d. How raauy new full-time faculty appointments have been made since July, 197 



^^Hcv many of your^ faculty members reported in (a), above, teach or supervise 



Full time 



Part time 



FTE* 



f. How many of the If acuity members who teach ot sup^vise doctoral su; ^..ts 
(e, above) ar'^ tenured? 

Full timr ^"t time 

g. How many of the faculty mrmbars who teach or supervise doetoral sUudfents 
(e, above) are chairman of ol e or more dissertation coOTittees? 

F ill time Pb. time 



ERLC 



*Ia this report p III is defined as the sum of full time plusfche full-ttoe 
equivelmt qc |iart»t4n^ faculty m^bei:& qmi ^ .u*'&ata. 
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2 



now Mr 



(e, above) have sarned Joccoif^es? 

Full tiffift 



tlirr 



f^^How ffmny of the faculty ttiembers w'no fien h or supervise doctoral students hav^ 
earned doctoiratea from departments your field at leading insLltutions 
aceording to "Quality of Graduate Faculty'' ratings in the ACE Roose- Andersen 
r^ort? (See liats at the end of thi^ form,) 

Full tim^ Part tlica 



FACUITY MSEAHCH ACTIVIIY 

How many facalty members who teach qt super^^lse doctoral Btud^nts are currently 
fiBttJed.ar least In paTt by a resear^hj project, or training grant or contract 
£rom some agency *.u*^HldB the univeys'tj^? 



Full tiae 



Part time 



is the total numbe^^ ou prof f^ssionril books and journal ar* xc .es by members of 
. CL ty (alone or In collaboration) publir.hed in th.t\ last three years (since 



Booki 



Jouttial artlcte^ 



Br I the last cnree years ^ ^low n^ny rasaaTch propoguls by f acuity h^riabara 

In ofepartment; ha^^a been submitted and have been funded i (1) intenia .ly 

(de?^i;^L€f.c or university; and (2) e?£terTaiiy (government, foundatltm, ur other 
oucsl^^ source) ' (Count prrsosals submitted to more than one agency or source 
only .nc«*) 

1972-73 
Source of Funds 
Internal External 



1973-74 
Source of Funds 
Internal Extf-rnal 



1974-75 
Source of Funds' 
Internal External 



a. Nuober of propojala 

^b* Total dollars of proposals 
utaltted (direct costs) 

c* Number of proposals funded 

a ^ot^ dolli o% projects 
funded C^irect costs) 



e.) ctual iollars available to 
fhe departoenf each year for 
resf^'^rch (Include university ^ 
"fle ^neyp" contlnuirti^ 
prj^Jects ^unded 1^ earlier 
years ^ and endowm^tot funds 
for r^. ^.;rch) 
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financial i:or e«.M oi tt)ii ^^^^ ■inrt.u vea 



Fall 1972 Fall 1973 Fall 1974 



a> PTE* faculty (inF^ructors tb.ough ' ■'hi>: i 

^ ¥TE* instructlonai support staff (adniAniy^- 
tratlve, office, technical, one.) 

a. ITE* iidergraduate departmeiital 

FTE* terminal ajtfir*s students 



ITE* doctoral students 



Baccalaurea*;e df^rDG^ iwarded 
Master's degree's rwardHd 
Doctor's degraea aw.'rded 

l-iDartmtint * t'^tal i( ' ir c r^\-r ? t ^ .^f, nn^i ^*en r-' 
ajtp^nse budiet for ea..n yRar, /aciuf mg ^^fts 
and utY r funds from outside t.^e ur ive?^^lty but 
apt ^nrludiT f stud€?nt f .lanciHl aid, assiPtant- 
ihips^ or re;2pHrch fund 

the approKlm^te dollar n". ^mt oT (1 ^ tliat was 
used to support: the. der r caien : ' h doc toral 
programCs/ . noc incluaing sti.H-r'L fii;ancial 
aidj asrLstantahlps, ct r^sea^ fimds 

OollatH (j) from soif-ee uut^.lJe i Ut; 
university <e,g,, glf'::. ^ c^^a^ ning ^r^nC support 
for fnculi s etc.p bit aut financial aid for 
studentf . ret:<^ar' i ^rdnts) 



July 1972- July 1973- July 1974- 
June 1973 June 1974 Ju ne 1975 



**Please hec'^^ the it fens in the following list that are included ^n your 
eaucation and genaral expense budget figure. 

^Inqtrucr ^ nal salaries 

Support staff salaries 



Fringe benefits 
^Supplias 

^Equipment , famishingSi and shop expenses 
Trav^ 1 

_Stua ; rec^u^'tMnt 
jConsultants 
^Library acqul. ' tions 



JComputer ti 
Janitorial aerviuca 
S\Hmaer school 



Other (please specify) * ^_ 



*TDcal of full til e plus full-time eq'lvalent of part time. 
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ERIC 



dccjotal study (or snterlns voar -rcitia) by those to whos y, ■ spri»d doctoral degrees In the last 



> 

65 

N 



llochiiicil 
Inorpnlc 
Or|inlc 
Pklieil 



IT 

Totil 

HISTORY 
Anclint 
Hidleval 



■world 
Other 

* Total 

PSYfflOlOGY 
ClklMl 
IducitlQQal 
bpiriiantal 
Divelopinul 
Hiasurei'^nt 
Or|gniiatio!i3i 
FersoDalley 
Soclil 
Othii! 

Total . ' 



Fall 19;i f .:ul:y 
to tsach doctoral 

Full tin Part lim 



Fall 19?4 doctor; . 
uil time Pirt tl e 



Doctoral degriea Aviri|a nuibtr of 
awarded years studanti taks 
My lin-June l^U ,o obtain dagnt 



ERIC 



m 



FuJJ time 



8, How many pcrsonB who r^ct-. ^ a io' tec's degree fro: your ciiepar tr^ent in 1973=1974 
had published a profcsiional . rticie, monograph, cr book prior to recGivlng the 
dmgTe a ? _^ ^ ^ 

i*, Bas^ on your experience In thp ist fe\. ye-rs^ what percent ige of enfering doctoral 
studants aventually complete t' e gree? 



10. .^DKISSIOHS DATA (Three ' s) 

How many students app led tor adTOlsslon to your 

doctprai prDgraniCs) in aach of the last three 
years? 

How many of these applicants ware offered 
admission? 

How many of theae adnl^^'^e;. i ndents actually 

enrc lied (reglstereu/i ? 

d, Ifhat was the average undergraJuate graae 
point average of the newly enrolled 

What were the average test scorfs of the 
^ewly enrolled doctoral students? f Include 
number of Individual icore^ Included in each 
avarage in parentheses following the score.) 

^ ORE Verl al 

^ i GEE Quantitative 

GRl advanced test In your field 

, Miller AnaJofles Test 

1, How m your newly enrolled doctoral 

studMts had already dompleeed a master 
dagree? 

J. How many of your newly enrolled doctoral 
students were U. S« cltliens from outside 
yarr acate? 



11. FINANCIAL SUPPORT FOR STUDQITS 



Fall 1972 Fall 1973 Fall 1974 



a. How many doctoral students held an s istantship in the department In eachJ^f the 
following categories durlr^ fall semester 1974? » 

Teaching ^Research ^Other 



EKLC 



i:,cj.uae scuaencs wi.o rect ivea reiLw^stap or rdinees^ ' Rfani ' r :ii ron-univer- 
slty aourcei, but do not Include Ic ins unle^=^ nhey wr^e part of a jnivf^rgity 
financial aid packaj|e. 



Whai LUe Lotai ooilar amounc c?r uni'"fiTs ity funas co your department for 
aasistantshlps or " ther work^relatiid doctoral studenr eupport ,i i^74-75? 



vmat ie the total acilar amount of uni'/crGlty funds to your dapartment fur non- 
work-raiated doctoral student support in 197^^-/^ (fellowships i grants, tuition 
remlssiony etc*-)? 

What Is the total dollar amount ^^f non-unlve ^ nity fuIlowshipSs grantSt and 
tr^lneaships to doctoral Frudants In ^^ou^ irttnent during 1974=75.^ 



mPLOYHmit 



What have been the ii st Jobs or doctoral degref^ reclplentM from our department n 
the last two yeara? Lount prlMry appointments If more than one j^b. The total 
number reported for eaci v^ar should equal Lne number of docc *s degrees awarded 
as reported In 5(h). 

Number of Doctoral 
Dagrea Recipients 

^72-73 1^/ i-74 

Postdoctoral fallowshi^* 

Regular ac^dCTlc ap ^InLtTient nt a FM granting unlvatslty 

Regular academic appoinrment at a fgur^year collegp nr 
non-PhD^granting university 

Lecturer or adjunct faculty appointmant at a four-year 
college or unlve'Slty 

Administrative position a four-year col^^ga cr univr'^'^ity 

Comunlty colltge position 

Prlm#-.ry or secondary school position 

Research or av^mlnlstration in business. Industry * or governfTient 
Professional practice In a clinic, agency, or hospital 
Self ^empT oye^ nr prlvi*^? ^ractlce 
.Other position 

uonCinuing graduate or p ^fessi nal education 

Seeking CTployment at last contact ^ * 
Unknc n 

Total ' ~ ~^ 

How many of these first Jobs were directly i elated to 

the graduate's field of specialisation? 

A84 ' / ^ 
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of data provided on the previous pages. Cn\ ies of recent, de-jartmental self- 
jtudieSj rfeport^ on isi-iueb such as admissions or trr . t= "raduart^s. 

and statements f procdurf ^ used to arrive at depa.. tmental sl. ioti wruld 
also be appre placed* 



Please return this ^'orm to your graduate ci^an or campTia -^oordinator for the 

COS Dimensions of - 11 y project, who will forward Ir to the project reseMrehers, 
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CHiMISTBY Phyilcjl Sci^i^i 



HISTOR — Social Sciencss 

i.e,iding instiUticni. by Rat# Quality A Grjduite Fieul^ 
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PSYCHOLOGY — Social Sc1«nctt 

Loading Inatitutlons, by Ratsd^Qyality of Gfadyatt Faculty 



iti7 


IBM 


iwaa 




s 


1* 


1 






i' 


2 


Mi.^iMMl 


i 




3 


Ctllferiilt. ilrtJli^> 


1 


1 " 


4 




7 


• 


i 


Milnela 




11* 


8 




8 


7* 


■< ' 


MiiwMula 


i 


7* 


7* 




3 


3' 


7*. 


Yi!« 




1 1 ' 


10 




1 ' 


1 

f 


11 
12* 


TiMS t 

irewn 


% 




12* 


AlJ.r 


t 


1 


14* 




12 


11* 


U 


Indisni 


i6 


li* 


Ifi' 




IS 


IB' 


16" 




t 


IS* 


18* 


NdfthwMMrn 


t 


18' 


Sum 1 


14 






Csmali 


i 


i' 


k" 


Iowa flew* ^hv) 


s 


23" 


20 


Mi^igin ^laM 


1 




fO" 


no^Mtar f 


i 


If 


24* 


&uk9 






24 


Nof^ Carelini 


1 


1 


24* 


Or*gsn t 


11 


14 


27* 


Celuf^bii 


21 


27* 








27- 








JO* 




1 


30' 




13 


20 


30* 





Dear rhairmai 



Four kinds or Inionnation a- ^^i^uqn^ ■ t^v-.: f^jrm: 

1, Reports r f currenc departmental prrcti^es in s=dj'ae"' ^te.j.. . 

2. Reports of recent changas t^cperienced b/ the departnient and your judgnients 
about their effect on the prograTn^s fiducational quality 5 

3* ^ur dtings of doctora ^lo^rams at a sampl',. 01 ; aiversities , 

^, Report and evaluation of depaTtm^i tal data cQllection for "his study. 

When you have completed the foiT^j pir^ise mail it .'irectly to Dr* Mary Jo Clark, 
Project Research Director, Educational Testin-^ .?«arvlce, Princeton ^ New Jersey, 
08540, . 

Yoiir nMe Da^te 



IJnl veraitv 



1 . Current de^artmeT "~ ?1 pr a^: ti^e£ P i_ - r. s e in d ... c a t e whe thar or not tne foil owing 

pra tices are cht^ractrti, c of vou ' ^eparCiTiejnt , by circiiiig one number on 

each line* „ ...^ 

Yes l2) 

No rn I 

a« There is a aaparto_entai tacuJ ty commi tte.^ tc mrnitor and i 

review; the doctoral progrra curriculmn, , 1 2 

b. The departtoent offers joint dcctoral legree programs with 

related departmr its , * - ^ . * . * , , . i 2 

Ca The department offers Interinstitutlonal or consortia 

prograffir at the doctorrl ii^vel* 1 2 

d. Th^e is a course or otrier systematic training program on 

- ^ ^= ^ ^ i -^'^^^ ^ — ^ r — p - ^ - ^ . — ^ 

aaaistants, . • • ^ . 1 2 

The department provides specific train lug for carefers 

o^hi. ^ than teach injg or r^s<^arcu ^ ..t. ..1 2 

f. There are regul c departmental procedures for the iValuatJou 

of courses and Inst. Lors by doctoral students*, . , 1 2 

g. The departoent mainUn.^ns placemen l files (recon^iendation& , 

vltae, etc*) for Its graduates. 1 2 

h* There is a departmental faculty committee to work on job 

piacfement of graduates , 1 / 2 



y ' AS'- 
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Hw frequently do the following practices occur 
In your department? % * * 



1. Faculty or experienced student "mentors" are 
^ assigned to worH with new teaching asalstants. 

j ^ faculty regularly visit claeseg^ being taught 
by teaching assistants.. * * 



Always (4) 
Ofteii (3) 
Sometimes (2) 
Never (1) 

i 

... 1 



• • . « 



k. Supervised field placCTents are arranged for 
students who want to prepare for careers pther 
than taachlng or research as part of the 
doctoral progrraa.. 

1, Individual professors obtain course evaluations 
from their doctoral students * * 

m. Students carry out independent research projects 
prior to reaching the dissertation sl ige 

n* A faculty member outside the 4eparttflent is 
appointed to 4 dissertation committee. 



. . . * 



2 ' 3 



9 * m 9 ' » * 



o. 



The dissertation commit tee^plays an active role 
In the supervision of a student *s graduate^ 
progrm. . . - • 



. • . . 



. . . • • 



r* 



Dissertation chalmen actively help their 
students locate appropriate positions upon 
completion of the degree. * • 

Th4 department encourages students and 
graduates to register with the university 
placement service. , . * * ' 

Most doctoral students attend colloquia 

by departmental faculty members or visaing 

scholars * * * * 



» . p * ■- 



... 



s. ,MDSt faculty attend' colloquia by departmental 
faculty members, or visiting scholars 



A88 



38i 



"3- 



if Recent: Ch anges , The left hand coiumn' below lists a number of changing conditions that, have been^ 
esperieneed by some graduate departments in the past few years, _ First, indicate whether each 
'condition has or has not occurred in your doctoral program in the last three" years* Next, If It 
has ©ccurred'i ^Ive your judgment of its effect on the academic ej^cellenca, o^ the prtgram, 

' . / ^ decurred in this If yes, effect bo far on 

doctoral program program's aeademlc 
* In last 3 vears _ exce Hence 



a* Decline in number of applicants for admission « . . • , 
b* Decline In nusijer of new students ad^iitted each year* , 

Ca Increase in number of dropouts and ABD's * ; 

d. Decline in university funds for student support * » *^ . 
e* Dec*line in external fellowship and tralneeshlp funds* * 
f* Decline. In sponsored research funds * » * * * • » - 

g, Declind in .funds ^for support services ifcid personnel , 

h. Decline Irt funds for supplies an^ equipment 

Is Redi/ced mnnter of faculty positions for the program , . 

J a More* difficulty placing new doctorates in appropriate 
academic positions . * , , ^ t ***** > * 

-. ' • e 

k. More difficulty placing new doctorates In appropriate . ■ 
* nonaca'demic positions s i .< ^ • 

1. Priiasure to develop more programs designed specifically 
to train students for nonacademic careers # • # . • * 

m* Pressure to hire minorities to meet affirmative actign 
requirements i*. 

n. Pressure to hire women to meet affirmative action 
requirements 

©.Pressure to admit minorities to meet equal educational 
opportunity requirements » 

p* Pressure to adtnit women to meet equal educational 

opportunity requirements • . i ****** 

q* Inflation of undergraduate grade point averages among 
applicants for admission 

r, Iiicrease in the vocational- orientation of etudcnts , , 



Yes (2) 
No (1) 
+ 

, , 1 



1, 



^ Improvement (4) 
No effect (3) 
Some damage (2) 
Serious damage (1) | 



2s*. 

2 , r 

2 \ * 

2 , , 
j ^ 

2 s s 

2's . 

1 , * 

2 . * 
2 , , 



* 9 m 9 * 



2 3 4 

2* 3 4 

2 3 

2 3 4 

2 3 ,4 

2 3 4 

.^.12 3 4 

\ 2 3 4 



, s 1 

. . 1 

s , 1 

s , 1 



1 2 3* 4 



, s 1 ^ 3 4 



12 3 4 



12 3 4 



12 3 4 



1 2 3 4 



12 3 4 



1 2 



3 4 



1 2 3 4 
12 3 4 
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3 For eaeli unlvcrolty' llBted below, rate the doccornl procramn in your field (dlsclplina) 
oS the CollowinB qucBUlons. Your ratinr.a will be used only in research associated wUh 
this projoctj lummaty ratiiigs will not b(s tnoue public. 

Ou^at lori Ai. Quality of Cfnduatc raculty.. Circle tho number below Che Cerm that cor= 
fcsjSr^dli^TorfTToTour judfimant of nhe qunllty of the craduatc, faculty in your 
• * fl«ld at each institution Mated. Consider only -the scholarly competence and achleve- 
•feenta of the preBont faculty." ■ . , 

. A.....tf..J... Of the noctoral Pranrnm: ffirclo tha number below t He term; 

, ^at eorr^Hponds tn^sT cloeoly to the way you would rate uha -Institutrons Ustod i you 
were solectlng a graduate , school at which to work for a ^doctorate today.. Take into 
%eeount^ the accessibility of the ^faculty and ite scholarly coinpetcnce the curricula, 
tminstructional and research facilities, the quality of graduate students, and other 
faetors that contribute to the effectiveness of the ^doctoral program, ^ 



INSTITUTIONS . 

(A sample of universities 
awarding doctorates in 
your field i arranged . \ 

- alphabetically) ' ' 



Boston College ■ 

Cal. at Berkeley j ^'1' 



Cal« at Bavis* U. of 

Cal. at Los Angeles, of 

Colorado, U, of 
faory Uf " 



Florida State U. (Taip.ahassee) 
Indiana U/ (Bloomington) 



Iowa, U/ of (Iowa City) 
Kansas » U. of _ _ 



4- 



Loulsiaua State (Baton RoJse) 
Maryland. of 

Hlchigan; U* of ' 
Minnesota, U. of 



Missouri', U, of (Columbia) 
New York U, 



QUESTION A 



QUALITY OF 
GRADUATE FACULW 



QUESTION B 



ATTRACTIVENESS^ OF 
DOCTGRM. PROGRAM 




Northwestern U# 

Oklahoma State U, (Stillwater) 



Pennsylvania, Ut of 
P r I'n ceton U. 



Stanford U. 

TQlQdo_3 U. _of_ 



Utah, U» of 
Vest Virginia U. 



Wrfconsin, U. (Madison) 



1 

.1 



3 
3 



4 
4 



5 
5 



1 
1 



2 
2 



3 
3 



4 
4 



4 
4 



2 



3 ^6 
5 6 



4 ^ 
4 



^6 
6 



.1 
1 



3 
3 



4 
4 



^5/ 
5 



6 
6 



6 
6 

6 



2 
2 



3 
3 



4 
4. 



6 
6 



7 
7 



6 , T 



1 


2 


3 


4 . 


S 


1 


2 


3 


4 


5 


1 


2 


3 


4 




1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


2 


3 ' 




5 


1 


2 


3 




S 




2 


3 


4. 


5 


^ 1 


2 


3 , 




5 


1 


2 




4 


5 


1 


2 


3 


4 


5 


1 


2 


3 




5 


1 


2 


3 


4 


5 


1 


2 


3 


4 


5 


1 


2 


3 


4 


3 


1 


2 


3 ' 


4 . 


5 


1 


2 


' 3 


4 


5 


1 


2 


3 ; 


4 


5 


1 


.2 


3 


4 


5 


1 


2 


3 


A 


5 


1 


2 


3 


4 


5' 


1 


% 


3 


4 


5 


1 


2 


3 


4 


5 


1 


2 


3 


4. 


3 


1 


2^ 


3 


4 


5 
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Report and evaluation of departmental 3ata collection for this study * 

We would' like to know how departments handled the on-'campus distribution of 
faculty and student questionnaires./ and your 'evaluation of thl^ procedure. 

Who managed the distribution and collection of faculty 
and stfudent questionnaires In your department? (Circlp 
one letter In each doiumn* ) 



Department chairman . - 
Faculty member * . . 
DepartmBntal assistant 
Secretaty i 
Student ^ . 
Dean's office , * . # < 
Other (specify) 



How were the qufestlonnaires distributed? (Circle as 
many letters as. apply.) 

Campus mail . . . « . ^ # « > * • 

U.S*'mail 'i 

Through classes i 
/ At »a faculty meeting 

At a. meeting of students « , . , 

Available to be picked up - ^ * > 

Other (specify) L 



Faculty Student 
Quest . Quest , 



c. "Did you a follDw=-up on non-respondents' 



F 

F 
F 
F 

F' 
F 



F 
F 
F 
F 
F 
F 
F 



F 
1. 



S 
S" 

s 
s 
s 
s 
s 



s 

'B 
S 
S 
S 
S 

s 



s 

s 



If ydu did a follow-up ^ what procedures did you use? 
(Circle as many letters as a{>ply,) . . ^ 

Verbal reminder In classes and/or meetings 
Memorandum r^lnder t • 

Distribution of a second questionnaire . * • 

Personal contact » ^ « « • « » « • 

V ^Otfej; (specify) __________ , 

In genera]., how effective do you think the distribution 
and colleation procedures were? - ' 

Very effective, everything went smoothly , , 
' , Fairly effective* although we encountered a 

few problems . 

Not effective; there were a lot of problems , 
Not effective because we had a limited staff 

and/or^ limited time • * * • * . • * s . . . < 

In general, what was the response of faculty members'^ « 
and students toward the research and the questionnaire? . . ^ 

Lrcle one letter in each column-) . ■ , ^ 

Enthusiastic 

Willing * 
Neutral ,,.,,*,.**,...-..*,•. 
• ' ' Reluctant ^ • • 

Resistant ^ .# • 



F 

F 
F 

F 
F 



F 
F 
F 

F 



F 

F 
F 
F 
F 



S 
S 

s 
s 



s 
s 

3 
S 
S 



g* How MLny faculty questionnaires did you, distributed 



EKLC 



How tnanyi student questionnaires did you distribute?' 
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One purpose of this project is to' assess the feasibility, of collecting a .wide 
variety of data relevant to- program quality. We would appreciate your comments : 
^and ,9U8gestlona donceirning the procedures and material^ used in this .study as well 
aa any problemB that have come to your. att,ention. . • ' , 



Pleaae cdmment. on the unique, inho vat ive, or PUtstandlng aspects of your dpctoral 
program (e.g., interinstitutional. or consortia arrangements , curriculim design, 
preparing graduates for non-traditional, employment opper tun.it ies , etc.). 
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Please conmant on t'he Rai'ticulaT inaQds or weaknespes of youp doctoral program. 



»• 

3 



Jhank you for completiiis thts form- and for your general Interesf in and support of 
this research. • . « v - • ' , • • 

ERIC : . ^ . : • . - ^Sb:^. ■ 



APPENDIX E 



Characteristics of Programs and 'Questidnnaire Reaportdet\ts 

Charactarls^ics of Programs in. the Study A97 
Background of Faculty Respon^i?nts A98 



Backgrouiid' of Student RespondentB A^*9 
Background of Alumni Respondents /^lOO. 



.r ■' ' ■■ 
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. - Table E.l 

Characterlatlcs of Programs 
(Average of program raports) 



Chamietry History Psyohology 
(N^21) . (N-23) (N^21) , 



1% Nimber of FTE 'graduate faculty _ 25 , ^ 28 ^ 29j ^ 

, 2 A Percentage of graduate faculty who . , 

are tenured 85 . 87 ^ ' 74 

3* 'Pereentage pt graduate faculty who » ^ ^ 

'chair dissertation cbimnittees * 84 .73 75 

4 / Number fe£ PTE doctoral students ,98 , 102 \ 119 

S, Nmnber of doctor's degrees awafded ' , , ^ ^ 

■* In a recent year \ ' 18 , 14 19 ^ 

'6. Ratio of doctors to baccalaureate " ^ 

' degrees , *62. ,11 *U 

7. Mean salary^prof essors " ' 24.5 * 23^1 23*4\^ 

'a. Maair^ salary— all faculty' - ^19.5 IS/O ^8^* 

9. Percentage of graduate faculty with ' S — 

ouf side research grant 68 10 T 37 1 

10, Outside dollars for research ■ > V ^ 

, (In^ thous^ds) - ^ 841 '48 . 468 

11, Research dollars per faculty ^ * ^ 
Cl%thousa^ids) V 31,0 1,8 ' 14.2 . 

12* Percentage of students "with financial . m ^ 

assistance ^ \ ^ ?1 54 84, ^ 

lJ,''Total financial aid dollars per, , 

, **student (in thouirfnds) / * 3,4 2*1 ^ 2*6 

14* Percentage 'of faculty with degrees = ... 

, from "leading institutions" _ ^ 84 91 72 ^ 



, JNotii jATerages for items 10 and 11 are based on returns from 
16 chemlstiyr departments, 1- history departments |i and 13 paychology*.^? 
departments. Item 13 Is based dh 18 19, and 48 responses respectively* 
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Table E.n 



Background of Faculty Respoadants 
(Average of program percBnts) 







Chemistry 


History Pay 


chology 




k 








1. 


Rank'i Professor 

Associate profefsor 


-^9 


54" 
30 , 


43 
28 


2. 


Ph.D. as highest earned degree 


^ |99 1> 


96 


.99 


3. 


Degree earned more than 10 years ago 


65 


51 ■ 


50 




Present appointtnent full tirne - . . 




yy 


OS 

yd 


5. 


Tenured appointment v^^* . 


81 


78- 


71 


6. 


Associated with this department mbre 
than 10 years ' ^ 


48 


33 


32 


7. 


More than 10 years of university' 

teaching eKperience^/ , ^ 

n 9 




5& 


45 


8. 


Joint appointment with another 
department * ' 


0,7 


t 

08 


11 


9. 


• 

More" than 50% of , time wit|i doctoral ' 
students 


" 46 




51 < 



10, More than^ 10 days away from campus 

for professional activities in the . . ^ 

past year , ^ 49 [ .29 ^ 44 

11* Three or more prof e^isiona^prlsenta- . , / 

tions in the past Vear*^,,^ ^57 35 " . 54 

'12* Outside research grants this year^ , 67 19 43 



13. Hlgheat degrees from leading 'Institu-- 

^ ttons" (peer rating 3,0. or above) ^ T 87 86 73 

14. Sex: male / ^ ^ 98 ' 92 86 

15. Average hours per week ini * /. 

teaching^ class preparation^ 

student evaluation 17 23 17 

b. advising students and directing , ' 

' rtudeht research T ' 13 ^ 7 10 

c#» ^search and scholarly writing 16 16 16 

d« progi-am and university admlAis-^ . * 

^ ^ration - ■ 1 ^ 7 7 7 

e* private 'praqtice - ^ ' ^ — X 

f . other professional activities 2 3 2 

^g, total pro'fesaional activities 56 ^56 55 
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Table E.3 



Background of Student Respondents 
\^ (Average of program percents) 







Chemistry 


History 


Psychology 


■1 


t ^ 

graduate degree and graduate ' ^ 
anrollment 


. 68 
^ I. » 


46 


- 

72 


•2. 


First enrolled in the program 1971 
or earlier (4 years or. more) 


58 ■ 


63 


53 


3. 


Degree expected in 1975 or 1976 


81 


,60 


75 


4. 


Degree goal of Ph.p* 


98 


98 


99 • 


5. 


Graduate sti^y only at this 
university ^ 


75 


* 50 


72 


Q • 


LfUrrentJL^ enroxieQ luxi Lxme 


92 


69 s. 


88 


,7/ 

i 


Undergraduate grades l 3*50 or above 
J / 3.00 ~ 3.49 


31 

48 f . 


32 ' 
45 ' 


47 

44 , 


8. 


r ■ ' _ . _ _ _ 

Graauate ^raoes jsju or aoove ^ 


55 


82 


80 


' 9. 


Currently research assistant 


47 


08 


21 


10. 


Currently, [teaching assistant 


3? 


33 


28. 


11'. 


>SeK- njale 




73 


65 


12. 


Age* under 30 


84 


56 ' 


80 


13. 


U.S. citizen 


82 


95 


93 




Race 1 Caucasian 

Black , ' 
Oriental' 


79 
10 , 


87 
04 
02 


90 

03 

02 
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Table E,4 

Background of Alumni Respondents 
(Average of 'program percenta) 





• 


Chemistry 
(N-20) 


Hlfltory 
(N-13) 


Psychology 
(N=14) 


1 » 


i 

Sbki male 


V 90 


88 


74, 


2. 


Agai 30-34 


66 


49 


59 


3. 


U,S • citizen 


a 9 




97 




"Racss CaucBsian 


89 

■ 


92 


96 


5. 


Undergraduate grades i 3 •50 or 

3*00 " 

m 


above 26 
3.49 49 


33 

52 


35 
50 


• 6. 


Graduate grades 3,50 or above 


57 ' 


76 


76 


7. 


Doctorate awarded i 1970 

1971 ' 
1972 


26 
33 , 
32 . 


27 
31 
33 


•24 
29 
40 


8. 


Employed full time 


98 


94 . 


91 




* Income $20,000 or over 


36 


11 


^ ^0 
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APPENDIX V 



Scales Formed from InJivldual Questionnaire Items 

Page 

Student Scales A103 
Faculty Scales Alll 
- Alumni Scales A117 



Notei EBtimates of scale reliability are 

intraclass correlations | coefficient 
alphas were computed to estimate 
scale homogeneity (Winer, 1971) * 



AlOl 



Student Scale No. 1. 
Invlronment for^Liarnlng ' 



F itm koyc 
of progrMi 
dlffamess 
4,16* 



Rsllibllitv of 
program sella 

■ . , ' ,76 ■ ' . 



Indix of , 
■ §cile , . 
hottogsneity' 

' .73.?::;' , 



~"- "^-"-> — r- ■ : f ' - ^ — ^ 

Itm'i^m correlitlons ■ of prpgrm ifean scores 


1 


2 


3, 


4 


5 6 : . . ^ 


L Comndn for students to feel exploited (reversed) 
■ ' (3-1=1), ! ■ , ■ ■ ^ - 




i 


= 






'2. Department has humane envlironment (S-l-l) 


61 








\ ■ , . ' 

! . 


3; Departmental practices create student tensions 
' * (reversed) (S^l-J) . ' 




■ 






■ ^ '■ . V- 


4. Coffipititlon'^amoni studints encowagad (revsrsid) 


33 


32 


16 




a 


5. Students support and help'each other/meet academic,. 
/ Uenands (S-l^q) '\ - 


06' 




,13 


i 

41 




" 6* Students fHendly and supportive in personal 

:= ' iivei (s-6-g) . i' ' ^ _ 


i 

02 ■ 


25 


05 


18 




'.' • • . ■ . ■■ ' ■ , ' ' ' ■ 
Item-scale score correlatlons^of prQ|ram mean scores". 


6« 


82' 


67 


62 





^^ ■,'*p.< .;001 d£ - 72/2025. 
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Student Scale No. 2 
Scholarly Excellinci of the Program 




FfwiAnova = ^ ^ . Riliabllitrof , ' . _ Mmoi ■ 

oi^^m^ . ' ' ^^^^^^ ' . ■ 

.88. ' ■ ' . /: ■ .90. 



dilittsncis 
8.60* 



1 2 3 4 5' 6 



IteHrltisa eorrilatlons of progrm mean scorts ' 
ii. Program m of bestln field (S-l-h) ' ' ' ■ T ' ; ■ 

a/'Studinti highly cofflpitent (S-l^n) , - ' _ '^^-^ 

■'■ . * . ■ ' - ' 

3, Department itiffiulating place to Jtud^ (S-l-t) ' , Bl ^ 50 

4. Faculty scholarship or ris^^ ' ' ^v , ' .. V ' 

^,8tu4€ntsdffflonstratfe originals ' ' 

reaearch (S-;64) ■ \ ;^53..60 S6 48 . 

Stidents C(»lttad tb puraiilt df scholarship^ (S^6-k) 49 SO 57^54 67; -. . 
jttffl-SMle score correlations of progrp mean scores 92 81 85 . ,88 75. 1\ 



1- 



*p < ,001 df ? 72/2W7 i 



'7'-. 



Student Scale No. 3^ 
Satisfaction wit!h Program 



: F. f roffi''Anova 
; of progt^ 
dif firtncis 
'3,47*' 




Riliabillty of 

^ progran scale 



scores 
/ .71 . 



Ite»-lt^ aprrelitlocii; of program itan scoris 
■ ■' L If had chaflci^; would' transf er ; (reversad) (S-l-d) 
Good preparation; for ay l^ter work (S-l-e) : ^ * 
; i. Would advise friend to attend here '(S-l-k) 
have4eai^e^ a SMat ieat (S-l-'o), 



^ 1 ,23' 4 



i2 



\ 74: 67 ■ - 



63 67 67 



Indix of \ 
, scale 

homogeneity 
.87,. 



Ittm^scale score corre^tions^f ^ograJii mean icores 



* ' 89 83 90 83 



.001 df = 72/2632 
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' ■ \ Student Scale No, 4 

I 

\^ . - V Faculty ConcMn for S/^udenti^^ 



F from Mm ■ Rgliability of • \ ' ■ Index of • 

of prograi * program scali , * . , sclli . 

dlfferineis ' , ■ / scores .homogeniity 

2,82* , .65. . * .80 



Ites^itim eomlatidns^of proiran ffl^^ 1,2 3 4 5» 6 7 

^...-r- \ - % ' . , . ■ ' ' ' 

■ 1, Acciisibility to. students (S*2-if — ■ \ 

; t '\ • • ,^ ^ , ; ■ ■ , 

' ■ 1 ■ ■ ' * . ■ . ■ ■ * = ■ ' ■ 

J. HilpJulKisi in idintlfylng ^ ,15 - 

3. Concern for professional devilopment (S-2-d) 
, 4, guidance In stlectlot of courses (S-2-e) 
5. Willlnlness to go out of way to help (S-2-f) 
J. Interest in students' welfare 

7. .Hilpfulneis in finding jphs after degree.' (S- 

, ;. -. _ " " . * ' \. 

-f 

• - - . T ' 

Item-'Scale s&re corr-elatlpns of program mean scoreg 



54 


35 






;;48 


-06 


52 - 




, 74 




65 53 






■A -. 






53 


15 


68 \47 


' 64 / ^ ■ 


:-m) , 18' 


,57. 


56 ir 


22 31 - 


eg 70 




87 -64 


74 76 65 



Student Scale Nb'.''5 
Rating Curriculum;^ 



F from Aiiova 
of progran ' 

dlfferancts 
4,89* 



Reliability of 
prograin scale 
scores ? 
.80 



Index of 

icale 
girisitj 

' .80 ' : 



Itim-itaB eorrelatlo|is of prpgrto mean scores 

Opportunitiis for courses In other fields (S-l-l) 
Z, Opportunities for Independent study (S-4-|) 



3. Flejtibility to mget individual niip (S-4-g) 



4. Sequince of courses (S-4-h) 
5i Brtadth of coursi and prog, offerings ,(sM-j) 
5^ Depth of course and prog, offerings (S'"4-k) 
7, Relations wlth^rilatsd depts, in-univ. (S-4-m) 

■'V- . ■ ^ i^. 

Itiir scale score cdtrelatlonii of program mean scores 



1 2 3 4 5 J 7 



56 72 ' ■ ' 

■04 00 10 - ' 

17 28 41 39 

23 31 43 42 , 85 - ' ■ 

42 23 46 32 ■ SO '58 . . - 

■ -m ' - ": 

58 63 81 42 75 79 74 



*p < Ml df - 72/2640 



) 



400 



F frm tawa. • ■ ' • ' , ^ Rallabllity of. , ; ■ ^f: 

of program , . • , ; program icale i , .fc™,'i!"4fi^' 



V Ittm-leem csmlitlons Of : prpgrM ffls^^ 1 ^ ^ ^ • /' 

f 2, Mevtnce of aMsslons riquirtfflM^^ (^H-b)- 53 :; " 

3. J^riaaent of degm re^^^ ■ . . : y ) 

4/Rilevanee o! riquirad QO^^ 31 34 46 » 

v 5. Apprefriattnesf of lai|uage riquirsmints (S-^i) ^ 04 04 17 0§ 



6, Fairness Of qMlifylijficam ato^ ' 31'* 33 62 46 . 23 - 

yrRil^mci Of rtquta^^^^ . 43 60 6? S9 17 ■ 51 



rtim-kale seore' wrrila^^^^ ' ^'^ 86. 62 ^46 . >73 79^ 




*p < .001 df niim 



Strident Scale' No. 1 
Quality of Teaching 

F froiaAaova , , ^ . Reliability of 

of, program * • ' ^ : - . vm^^ scale 
dlfftreness ' ^corts - 



.74 



ttp-itOT eorrilationg of program^ 
.ilxMany faculty do not pripar(a.for cQuries (rivtbid)' 



^ *p < .Q01* df ^ 72/2618 



2 3 



L Useful f acuity aritlcism of my work (^^^ ■ 34 

■ , ^ . _ . ^ # : j . -. ^ . ' _ 

J. Hilpfulnass in dealini with classwork (S-2-g) 35 ' 53 — 
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. Faculty Seals No, 1 
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Resewch Activity 



I f EQm Anova 
difftriocas 



Riliabillty of 

progr^ scale 
' ^scores ' 
.83 



IndsK of 

'sella , . 
homogenilty 

.68 



ifiet^lt^ eorrilations of progrim mta^ ' 


1 2 3 




1 


' 1, keiiareh/wTltlng award (I-8-i) 








■ '2. Journal iiitotf ' ' 


■ 17 - 






3. Article refer ae (1^8- j) 


25 1 32 ™ 






4, Instltutional/departfflintal grant (F-8-k) . 


16 13 36 


i 




5. Outsld^gknt/ contract (F-8-1) ^ ,f 


32 22 70/ 

f * ■ 


' 28 - 


i 


Item-scale scort corralatlons of, program mean scores 


52 51 82 


51 m 





V*p < .001 df ^ 72/1679 ' , , 

Npter "Scale, for. this item Is average nmnber.of 
I. "yes" responBiS by faculty members in 
^ each program, 



: ERJC 



407 



' .4 : ' ^ FacUty Scale No. 4 ' 

. . ■ Rating of Studant Comitinent/Motivation 
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Alumi Scale fo. 1 
Satisfaction with Dissertation EKpsri'encfiS 
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Examples of Feedback Reports to Individual Pro|rms 

' ' ' ' Page 

Contants of Feiadback Reports A121 

Sample Pages of Feedback Reports " A123 

.Interpretive Guide for Questionnaire Feedback ' A125 

^ Larger-smaller departmental groupings- and 

number of ' respondents - for chemistry 5 history 5 

^and psychology , ,A129 



A119 
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I 'CGS/ETS Graduate Department Study 



■ S"'-' ■ ■ / = 

s I Contants of Feedback Repprts 

Student Queatlonn^rs 

^^A, Student perceptions of doctoral program academic envlronmant 

B. Student perceptions of intarpersonal environment and 
overall satisfaction 

C* . Ratlnge of faculty teaching and scholarship , ■■ ■ - 

Dt Ratings of faculty concern *for_ Students - 

E* Ratings of departmental facilities^ 

F, Ratings of program offerings and curriculum 

G, Ratings of degree requirements and program .procedures 

H* Ratings of ^departmental teaching and research asslstantships 
(based ^ly on responses pf assistantshlp holders) 

I* Characterization of doctoral students In the department ^ , 

J. Importance of three program purposes - \ ^ 

K* Background of respondents 

Faculty Questionnaire . 

A* Faculty perceptions of doctoral program -academic environment 

B, Faculty perceptions of interpersonal environment 
Faculty teaching and schbliirshlp 

P* Ratings of depattmental facilities-, 

^ E* Ratings of degree requirements and program procedures.^ 

P, . Faculty morale and status - 

G* Faculty professional involvement % , 

H, Characterization of doctoral students In the department 
I* Importance of three program purposes 

J* Characteristics of respondents 

K, Professional time commitments (see also "N'S below) 

L* Public€tlons 

M.' Average number of journal publications 

N, Average time per week in professional activities 

0, (1) Colleague ratings of doctoral progipam (tabulation of 
individual riaspondents) 

(2) Colleague ratings of program fields of specialization 
(tabulation of respondents by spec^lallzation) 
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A* Alifflni perceptions of doetoraX program academic envirorunerit 

B. klmmi. perceptions of interpersonal envlroiment ^ 

I. 'C# /Ratings of faculty teaching and scholarship 

D* 'Ratings ot faculty concern for students 

. E, Ratings of , program offerings and curriculum 

F. Ratings| of degree requirements and program procedures 

G. . Evaluation , of dissertation components , ^ 

H. ^, Sources of job placement help 

I. Employment , - ' ^ ' ' 
Jp , Characteristics of current pogition ' 

^ K. Value pf graduate experiences for present work 

L. Research » 'publications ^ and presentations 

M, Background of rjespondents 
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CGS/ETS Graduate Department Study 
Interpretive Guide, for Questionnaire Feedback , 

Eaeh department should receive at least two separate feedback 
reportS"£Qr student and faculty respondents-^-whlch are computs^ sunrniariea 
of the responses of these groups to each item on the questionnaires they 
coi£q>letsd last, spring. (Some departments will also receive a feedback 
report for their alunmi. If your department has not received one of 
these, it Is because not enough responses had^ been received from alumni 
of your department in time to process them.) A two--page table of contents 
for these summaries (with a page index) is provided for your nonvenlence* 

^ The first Important thing to bring to your attention about^ the 
computer repotts, is that the item summaries are not in the same order 
in whiph the items appear on the questionnaire. Instead,' the items are 
grouped acQording to general content similarity so that responses that 
deal with similar aspects of the department are together. On ths student 
simiary, for example, the first group of items that are reported are 
those; that have to do with "Student Perceptions of Doctoral Program 
Academic Environment" (which includei a suBmary of student responses to 
questionnaire items l^a, 1-b, l-*e, 1-g, etc.); the second group of items 
(beginning on page 4) have to do with "Student Perceptions of Inter-- • 
personal Environment and Overall Satisfaction" (which consists of ques- 
tipimalroi Items 1-d, 1-*!, l^j, and l^k) ; and so on. There are 11 such 
content categories for the student report (item ;^rnuplnga A through K) , ^ 
15 for faculty, and 13 for. the alumni. The' actual questionnaire item 
ntutiber precedes each item stem on the^ report. \i ' v 

/M-^f % . . . . ' 

Note that the wording of the item stems Vhave been abridged on the 
computer report* Becaufe of this, it mlgn^'be well to go over the report 
with the original quf stionnaire in^hand, so that jou'have the complete 
and accurate wording of the questions. (An extra copy of the questionnaire 
is included with each report. ) 

Mobt of the sumoaries have three columns of numbers. (Exceptions 
aire the reports for alumni in history and psychology departments. See 
comments under Alumni section below.) The first coltuoLi consists of data 
for your omi department, with these data based on the niraber of individuals 

tad at the top of the colmm (under the words "your dept.")* The ji™ber 
and percent of individuals .in your department who chose each response 
"Option are listed. The second and thir4 columns are provided so that you 
can compare the responses of persons in your ami department with' those at 
other larger and smaller departments in the study in your part IcUlar 
discipline* The numbar of departments comprising each group. Is Indicated 
iunder the coluim headings "larger depts." and "smaller dapts." ^ (The 
asslgnmant ;of a particular department to the larger or smaller group was 
djOne by simply taking recent data regarding the numbar of Ph.D.'s produced 
by aach dapartmant, rmk^ordaring, the dapartments on the basis of those 
figures p and dividing at the median or middle point* The names of tha 
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speeiflc dep|rtmanC& making up the larger and smaller comparison groups » 
^ dlSQipllnep are provided on the ^.accompanying sheet.) Only percent 
llgures are reported for the two comparison groups and, in both cases » 
' thass are averages of pereents at the number of 'departments Indicated 
in the colutm heading (usually 12 or 13) . 

in comparing your department's data with those of other departmental 
it ie important to remember that the percentage figures refer to percent 
of those (studentifei * faculty or alumni) who returned a useable question- 
^ naire* Overall, the response rates across departments were quite high 
especially for students and faculty; that is, at most departments a very 
. high percent of the students and faculty who were asked to complete 
f questionnaires did so» But there were some^ eKceptlons to this general 
rule. The number of useable responsee from each department are Included 
with the list of departments in esfch comparison group. 

.A final comment is that the percent ^^slgnated "other" on the suimiiary 
sheets is simply the percent of respondents who omitted that particular 
Itfem. ' 

■• ^ ■ ^ ' . ' 

Besides these genpral interpretive guidelines there ar^variations 
^in the computer sununaries for students, faculty, and alumnii that probably 
need to be clarified, A specific discussion of each of these follows. 

Student summary , Aa-^with the other reports, most percents in the 
student summaries refer to percent of students completing the questionnaire 
(that la, the number listed under the wbrds ^-youv dept." at the top of the 
first column). However, there is an important exception to this general 
rule, namely the student ratings of departmental asBlstantships (Section H 
on page 20 of the student summaries). Because only students who held an 
assistantship (teaching or research) $t one time during. their graduate 
student experience were asked to complete this section, percentagss are 
based on the numbet^ of persons, who answered each Item rather than the 
total number of questionnaire respondents. 

Fabul t3^s unma ry , As with the other suraaariesi most responses to the 
faculty questionnaire are reported in percents based on the total number 
of respondents in each program. Two exceptions on the. faculty sumary, 
however, pertain to Sections M and N ou page 22, For these items, the 
mean (arithmetic average) number of publications (in M) and mean nimber 
of hours per week spent on various activities (in N) are reported*" 
^(Numbers under the "SD" column refer to the standard daviatlpn of the 
same numbers. The standard deviation Is a statistical expression having 
oto do with the range of .observations—in this case, publications and 
%tlme spent-*-and does not need to be considered in any detail here,) The * 
ind^x of aAnual productivity of articles and book reviews (Item 9-^A*l ^ 2 
under Section M) was obtained by ^taking each respondents* entire career 
artlclas', book chapters, and reviews (as siported in Items 9-^A-l and 
,9-A«2) and dividing by the number of years alnee that individual obtalnad 
J his or her* highest degree (which in the gteat majority of cases, of caurseV 

was the doctorate)« 
' =. ■ ' ■ ■ 
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One other section of the Eaculty summary that is somewhat different 
than the other sections la Section 0| on pages 23-*25, having to do with 
colleague ratings of your doctoral program. First, It should bo noted 
that overall ratlngi of your departiimnt (itema 25-=A and 25-=B) are provided 
for both your own faculty and those at other departments » but ratings of 
your departments- specific fields of Bpeciall^atlon (pages 24 and 23) are 
provided only for faculty from the other institutions In the study. The 
second way in which the data in^this section differ from the rest of the 
faculty summary p is that the percents iu Saction 0 are ba^ed on individual 
faculty members rather than being thQ average of the percents across" the 
dozen or so departments in the larger or smaller category. And finally, 
the number and percents for each specialization rating represent only 
responses by faculty at other institutions who Indicated the same area of 
specialization as the one they were rating. (See item 26 on page 9 of 
the faculty questionnalra for clarification. ) Thus, these percents were 
computed on a varying nianber of faculty membersi that nt^ber being the 
one &n the line labeled "total in specialization' who responded*" 

M.\mni_ auimary . Two important points need to be made about this 
summary » both having to do with the fact that many fewer questionnaires 
were returned from alumni than was the case for faculty or current stu* 
dents« first I you' may not have received a summary for your department -s ^\ 
alutttil at all. Alunmi summaries were not run for those departments with 
fewer than 10 returned alumni questionnaires by the time these materials 
needed to be processed* (In cases where late returns hava^very recently 
boosted the number responding to more than 10, we will send altuonl sum- ' 
marles at a later date.) Second, the comparison data for the ali^nl 
suimDariea is different In two respects. One is that onjy departments 
with more than 10 alumni respondents were included In comparison groups, 
(see comparison group sheets). Therefore » a smaller number of departments 
are included In the larger and smaller comparison groups for chemistry. 
The other difference is that for history and psy;chology large and stjiall 
departments have been combined Into one comparison group since only 
13 history and 14 psychology departments had sufficient returns. . 



' ' , ^ General Information and Next Steps 

We hope that you will find these departmental surmarles Interesting 
and usefuli They represent the first step in the data analysis md .report- 
writing phase of the project. During the' next several months we shair be^^ 
sifting^ so^'tlng, and exploring these data iri a variety of ^ ways as we 
search for a better understanding of the complex relationship between the ^ 
reputation of an academic department as perceived by those outside of the 
department on the basis of the v'isibillty of the faculty i level Of rtfsearch 
productivity t and the like » and the inner life of the department (faculty 
moralSi level and fr^equency of student--f acuity interactions, and so on)* - 
The final reporjt should be completed by early 1976s at which time we shall 
send a siumary^ to; you. M * / ^ 
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Uppermost in our minds during this process will be. the joint questions 
of feasibility and utility* That ISp no matter what the results of our 
data'-analyals phase of the project* we must ultimately aak how useful such 
Inforpiatloii will be to those involved In graduate education—particularly 
at the dapartmsnt Jsvel" and how feasible Is It to consider the collectl^on 
and analysis of such Information on a recurring basli* In order to answer 
these quastlbns we^wlll be Imposing on some of you just one more time 
dating .the next few weeks. ^ Either by phone ^or direct personal ylsiti we 
would like to ask some of you to give us' your reactions to the questions 
posed abpVe regarding the^ feasibility of the procedure your department 
h|is taken part in, and the utility of the resulting infomafclon* 

In the meantime. If you should have any questions that cannot be 
answered by your institutional coordinator In the graduate dean*s office, 
or ^^f youj have any "coraaaents or criticisms about the project in general, 
do not hesitate to contact us by phone or mailt 



Drs. Mary Jo Clark (Project Director), Rodney T. Hart nptt, or 

Leonard L* Baird . , . 

Developmental Research Division ^ 
Educational Testing Service 
Princeton, New Jersey 08540 

.jC609) 921-9000 
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CGS/ITS Graduate Department Study 
Comparison Groups for CHmiSTRY 



Larger Departments 



Unlv . of Calif * at Berkeley 

Univ. of Calif, at Los Angelas 

Univ. of Colorado ' 

Florida^State Uni^* 

Indira ^Unlv. 

Univ* of Maryland 
^Unlv* of Michigan ' 

Univ. of Minnesota 

Nor thwes tarn Univ. 

Princeton Univ* 
.Stanford UniVf ^ . 

Univ* of-Wisconsih ^ 



' Smaller Departmante 

Boston Collage . 

Univ* of Calif, at Davis 

Emory Univ. 

Univ. of Iowa 

Univ. 'of Kansas 

Louisiana State Univ# 

'Univ* of Missouri 

New York Univ. . 

Oklahoma St'ate Univ. 

Univ. of Toledo 

Univ. of Utah 

West Virginia tfalvi 



t 



Number of Res 
Faculty Student 



ents 

Alunml 



38 
26 
24 
26 
21 
24 
18 
22 
26 
20 
16 
36 



12 
21 
11 
18 
20 
30 
16 
17 
17 
8 
24 
20 



90. 
46 

35 
31 
52 
35 

22, 
23 



35 
44 
104/ 



6 

le" 

21 
27 
15 
,23 
11 
26 
27 
8 
34 
14' 



' 11 
23 

23 
17 
11 
20 
10 
32, 
" 23 
25 
, 66' 



20 

23 
19 
24 
10 
15 
12 

11 
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CGS/ETS eraduate Department Study 
Comparison Groups for HISTORY 



1. 

Larger Departments 


Nimber 
Faculty 


of Reapor 
Student 


idents 

Alimnl 


UnlVi of Calif * SLt Berkeley 


30 


52 


- — 


\ Univ. of Gallf. at Los Angeles 


38 . 


^72 




Univ. of Colorado 


23 


14 


-- ■ 


Indian^ Univ. . . 


28 


53 


30 


TTfl^V. f%f MlrtiicPAn m 








Uniy^i of Minnesota \ 


36 


38 


• 12 


UniVi of Missouri 


18 


31 


18 


New York Univ. 


16 


58 


30 


Nbrthwestern Univ. 


18 


36 


19 


Princeton UniVi • ' 


37 


25 


18 ' 


Stmford Univ. 


24 • 


45 


25 


Univ. of Wisconsin 


33 


124 


40 

■ 


Smaller Departments ' 




. ■ - 




Boston Colibga 




9 • 




Univ. of Calif at Davis ^ 


27 ' 


29 




Emory Univ. 


' 16 * 


16 




Florida State Univ. ' 


25 


23 


— 


Univ. of Iowa ; 


22 


34. 


11 


Unlv; of Kansas 


30 


22 


14 


Loulsiana^State Uhlv. 


19 


23 




,Unlv* Maryland 


29 


37 


11 . 


Oklahoma State Univ. : 


14 


21 




/Univ* of Pennsylvania 


. 15 


19 




Univ. pi Tdlado 


10 


.11 




Univ. of Utah ■ / ^ 


19 


27 




West Virginia Univ,. . 


,18 


9 





\ 
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CGS/ETS Graduate Depirtment Study 
Comparlsori Groups for PSYCHOLOGY 



Larger Departmenti 



.-0 



UniVi^of Gallf, at Berkeley 
Unlv, of Calif, at Los Angmlm 
Unlv* of Colorado 
Florida State Univ. , . 

Univ. of Iowa 
UnlVi pf Kansag 
Unlv> of Miehigan 
Univ* of Minnesota 
UniVt of Misaourl 
New York Unlv* 
Stanford Univ* 
Univ* . of Wisconsin 



Ntmber of Reepohdents 



Faculty 



16 
41 
34 
39 
26 
32 
61 
59 
17 
18 
22 
22 



Student 



19 
•• 30 
82 
74 
43 
47 
33 
202 
26 
93 
34 
48 



Alumil 



,11 
30 

33 
'25 

35 
20 
19 
22 
33 
17 



Spaller Departments 
Bostdn College 
Univ. of Calif, at Davis 
Emory UnlV. 
Indiana Univ. " 
Louisiwia State Univ. 
Univ. of Maryland 
Nprthwestem Univ. 
Oklahona State Univ. 
Univ. of Pennsylvania 
Univ.* of Toledo 
Uniy. of Utah : 
Wrist Virginia l^v. 



15 
17 
18 
20 

18 
32 
15 

16 
11 
21 
19 



12 
11 
22 
23 
24 
37 
15 
17 
23 
6 
28 
18, 



17 
17 
15 



13 
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